Hi,
I''ve been touring XCP and making my way around it for the last few
days, and am having problems with some reasonably straightforward
actions. I''ve poured a several hours into different ways of resolving
the issue, but am coming up short.
Here''s the main issue. The following command (plugging in a VBD to
the dom0 VM), when, run, hangs for several minutes, and then produces
the following output:
$ xe vbd-plug uuid=66b1f602-fe0b-f4f7-ba41-47c7f7e01dea
<SITS FOR SEVERAL MINUTES>
A timeout happened while attempting to attach a device to a VM.
type: VBD
ref: 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea
The VBD in question is for a VDI that exists on a Local disk SR of
type ''ext'' (i.e. mounted ext3 disk). The SR is attached and
mounted,
and everything seems OK there.
After the command, there''s a ''xvdb'' device that shows
up in dom0,
which seems to indicate that the attach was successful (at least at
the low level, even if not within xapi):
[root@localhost ~]# ls -l /dev/xvdb
brw-r----- 1 root disk 202, 16 Apr 15 13:45 /dev/xvdb
And tap-ctl lists the attached data disk:
[root@localhost ~]# tap-ctl list
4010 0 0 vhd
/var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd
I was able to make an ext3 FS on xvdb and then mount it, and confirm
that the tapdisk2 process associated with the device was working
properly. That said, the listing according to "xe" shows that the VBD
is not "currently-attached".
I''m somewhat perplexed now as to what the issue might be.
I''ve
included the dumps for all the relevant objects at the end of the mail
(last bits of the SMlog, short listing of VM object, long listing of
VBD, VDI, SR, and PBD objects, the output of "tap-ctl list", and the
mounts in dom0)
Thanks for any help.
Cheers.
-kannan
Relevant info:
The SMlog has the following at its tail:
[3973] 2011-04-15 13:44:04.806595 [''/usr/sbin/tap-ctl'',
''list'', ''-m'', ''0'']
[3973] 2011-04-15 13:44:04.824135 = 0
[3973] 2011-04-15 13:44:04.825200 tap.activate: Launched
Tapdisk(vhd:/var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd,
pid=4010, minor=0, state=R)
[3973] 2011-04-15
13:44:04.826014
DeviceNode(/dev/sm/backend/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db)
-> /dev/xen/blktap-2/tapdev0
[3973] 2011-04-15 13:44:04.826896 lock: released
/var/lock/sm/2463c214-05a9-49d4-8cba-b60ceb6b16db/vdi
[3973] 2011-04-15 13:44:04.827179 lock: closed
/var/lock/sm/2463c214-05a9-49d4-8cba-b60ceb6b16db/vdi
[3973] 2011-04-15 13:44:04.827743 lock: closed
/var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr
[3973] 2011-04-15 13:44:04.828687 lock: closed
/var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr
[4022] 2011-04-15 13:44:05.047143 sr_content_type {''sr_uuid'':
''17369039-53e9-5c0f-9e27-141a74b1a50d'',
''subtask_of'':
''OpaqueRef:caeed889-30bc-1c49-b634-820feeecf677'',
''args'': [],
''host_ref'':
''OpaqueRef:e41042ed-25cb-1a7c-cbe9-56a110e2440a'',
''session_ref'':
''OpaqueRef:97a2a6fd-2ca7-da1c-d540-719b0a393277'',
''device_config'': {''device'':
''/dev/hda3'', ''SRmaster'':
''true''},
''command'': ''sr_content_type'',
''sr_ref'':
''OpaqueRef:0a66472b-1020-b868-801d-f83e0ff2a18b''}
[4022] 2011-04-15 13:44:05.047689 lock: closed
/var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr
This is the state of the log a just few seconds after the command is
run, and there are no more log lines added after this. The log sits
like this during most of the command''s run, and even after the command
finishes with an error.
The relevant objects:
VM (dom0):
[root@localhost ~]# xe vm-list
uuid ( RO) : dd6681cb-f886-4847-b267-a24f555e14ee
name-label ( RW): Control domain on host: xcp
power-state ( RO): running
VBD:
[root@localhost ~]# xe vbd-list params=all
uuid ( RO) : 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea
vm-uuid ( RO): dd6681cb-f886-4847-b267-a24f555e14ee
vm-name-label ( RO): Control domain on host: xcp
vdi-uuid ( RO): 2463c214-05a9-49d4-8cba-b60ceb6b16db
vdi-name-label ( RO): testvdi
allowed-operations (SRO): pause; unpause; plug; attach
current-operations (SRO):
empty ( RO): false
device ( RO): xvdb
userdevice ( RW): xvdb
bootable ( RW): false
mode ( RW): RW
type ( RW): Disk
unpluggable ( RW): true
currently-attached ( RO): false
attachable ( RO): <expensive field>
storage-lock ( RO): false
status-code ( RO): 0
status-detail ( RO):
qos_algorithm_type ( RW):
qos_algorithm_params (MRW):
qos_supported_algorithms (SRO):
other-config (MRW):
io_read_kbs ( RO): <expensive field>
io_write_kbs ( RO): <expensive field>
VDI:
[root@localhost ~]# xe vdi-list
uuid=2463c214-05a9-49d4-8cba-b60ceb6b16db params=all
uuid ( RO) : 2463c214-05a9-49d4-8cba-b60ceb6b16db
name-label ( RW): testvdi
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): resize_online; update; resize;
destroy; clone; copy; snapshot
current-operations (SRO):
sr-uuid ( RO): 17369039-53e9-5c0f-9e27-141a74b1a50d
sr-name-label ( RO): Local File SR
vbd-uuids (SRO): 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea
crashdump-uuids (SRO):
virtual-size ( RO): 1073741824
physical-utilisation ( RO): 5120
location ( RO): 2463c214-05a9-49d4-8cba-b60ceb6b16db
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO): <not in database>
missing ( RO): false
other-config (MRW):
xenstore-data (MRO): scsi/0x12/0x83:
AIMAMQIBAC1YRU5TUkMgIDI0NjNjMjE0LTA1YTktNDlkNC04Y2JhLWI2MGNlYjZiMTZkYiA=;
scsi/0x12/0x80: AIAAEjI0NjNjMjE0LTA1YTktNDkgIA==; vdi-uuid:
2463c214-05a9-49d4-8cba-b60ceb6b16db; storage-type: ext
sm-config (MRO):
host_OpaqueRef:e41042ed-25cb-1a7c-cbe9-56a110e2440a: RW
on-boot ( RW): persist
allow-caching ( RW): false
tags (SRW):
SR:
[root@localhost ~]# xe sr-list
uuid=17369039-53e9-5c0f-9e27-141a74b1a50d params=all
uuid ( RO) : 17369039-53e9-5c0f-9e27-141a74b1a50d
name-label ( RW): Local File SR
name-description ( RW):
host ( RO): xcp
allowed-operations (SRO): forget; VDI.create; VDI.snapshot;
plug; update; destroy; VDI.destroy; scan; VDI.clone; VDI.resize;
unplug
current-operations (SRO):
VDIs (SRO): 2463c214-05a9-49d4-8cba-b60ceb6b16db
PBDs (SRO): d01bc58b-ad8d-3986-b5ba-7a75cad0d89e
virtual-allocation ( RO): 1073741824
physical-utilisation ( RO): 196935680
physical-size ( RO): 97221701632
type ( RO): ext
content-type ( RO): local SR
shared ( RW): false
other-config (MRW):
sm-config (MRO): devserial:
blobs ( RO):
local-cache-enabled ( RO): false
tags (SRW):
PBD:
[root@localhost ~]# xe pbd-list
uuid=d01bc58b-ad8d-3986-b5ba-7a75cad0d89e params=all
uuid ( RO) : d01bc58b-ad8d-3986-b5ba-7a75cad0d89e
host ( RO) [DEPRECATED]: 8f1f205a-d352-48cf-afa1-7f2fe1f0f7cf
host-uuid ( RO): 8f1f205a-d352-48cf-afa1-7f2fe1f0f7cf
sr-uuid ( RO): 17369039-53e9-5c0f-9e27-141a74b1a50d
device-config (MRO): device: /dev/hda3
currently-attached ( RO): true
other-config (MRW):
TAP-CTL LIST OUTPUT:
4010 0 0 vhd
/var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd
MOUNTED FS in dom0:
[root@localhost ~]# mount
/dev/hda1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/mapper/XSLocalEXT--17369039--53e9--5c0f--9e27--141a74b1a50d-17369039--53e9--5c0f--9e27--141a74b1a50d
on /var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d type ext3
(rw)
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Figured out what was happening. Turns out the udev scripts in the guest weren''t getting invoked properly and kicking the appropriate xenstore paths, so xapi was hanging on attaching disks since the completion event from the guest would never get delivered back. Right now I''m just manually watching the xapi logs and kicking the paths myself.. but it should be easy enough to fix now that I know the issue. Thought the resolution might be helpful for others. Cheers. -kannan On Fri, Apr 15, 2011 at 2:35 PM, Kannan Vijayan <kvijayan@gridcentric.ca> wrote:> Hi, > > I''ve been touring XCP and making my way around it for the last few > days, and am having problems with some reasonably straightforward > actions. I''ve poured a several hours into different ways of resolving > the issue, but am coming up short. > > Here''s the main issue. The following command (plugging in a VBD to > the dom0 VM), when, run, hangs for several minutes, and then produces > the following output: > > $ xe vbd-plug uuid=66b1f602-fe0b-f4f7-ba41-47c7f7e01dea > <SITS FOR SEVERAL MINUTES> > A timeout happened while attempting to attach a device to a VM. > type: VBD > ref: 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea > > The VBD in question is for a VDI that exists on a Local disk SR of > type ''ext'' (i.e. mounted ext3 disk). The SR is attached and mounted, > and everything seems OK there. > > After the command, there''s a ''xvdb'' device that shows up in dom0, > which seems to indicate that the attach was successful (at least at > the low level, even if not within xapi): > > [root@localhost ~]# ls -l /dev/xvdb > brw-r----- 1 root disk 202, 16 Apr 15 13:45 /dev/xvdb > > And tap-ctl lists the attached data disk: > [root@localhost ~]# tap-ctl list > 4010 0 0 vhd > /var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd > > I was able to make an ext3 FS on xvdb and then mount it, and confirm > that the tapdisk2 process associated with the device was working > properly. That said, the listing according to "xe" shows that the VBD > is not "currently-attached". > > I''m somewhat perplexed now as to what the issue might be. I''ve > included the dumps for all the relevant objects at the end of the mail > (last bits of the SMlog, short listing of VM object, long listing of > VBD, VDI, SR, and PBD objects, the output of "tap-ctl list", and the > mounts in dom0) > > Thanks for any help. > > Cheers. > -kannan > > Relevant info: > > The SMlog has the following at its tail: > [3973] 2011-04-15 13:44:04.806595 [''/usr/sbin/tap-ctl'', ''list'', ''-m'', ''0''] > [3973] 2011-04-15 13:44:04.824135 = 0 > [3973] 2011-04-15 13:44:04.825200 tap.activate: Launched > Tapdisk(vhd:/var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd, > pid=4010, minor=0, state=R) > [3973] 2011-04-15 > 13:44:04.826014 DeviceNode(/dev/sm/backend/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db) > -> /dev/xen/blktap-2/tapdev0 > [3973] 2011-04-15 13:44:04.826896 lock: released > /var/lock/sm/2463c214-05a9-49d4-8cba-b60ceb6b16db/vdi > [3973] 2011-04-15 13:44:04.827179 lock: closed > /var/lock/sm/2463c214-05a9-49d4-8cba-b60ceb6b16db/vdi > [3973] 2011-04-15 13:44:04.827743 lock: closed > /var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr > [3973] 2011-04-15 13:44:04.828687 lock: closed > /var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr > [4022] 2011-04-15 13:44:05.047143 sr_content_type {''sr_uuid'': > ''17369039-53e9-5c0f-9e27-141a74b1a50d'', ''subtask_of'': > ''OpaqueRef:caeed889-30bc-1c49-b634-820feeecf677'', ''args'': [], > ''host_ref'': ''OpaqueRef:e41042ed-25cb-1a7c-cbe9-56a110e2440a'', > ''session_ref'': ''OpaqueRef:97a2a6fd-2ca7-da1c-d540-719b0a393277'', > ''device_config'': {''device'': ''/dev/hda3'', ''SRmaster'': ''true''}, > ''command'': ''sr_content_type'', ''sr_ref'': > ''OpaqueRef:0a66472b-1020-b868-801d-f83e0ff2a18b''} > [4022] 2011-04-15 13:44:05.047689 lock: closed > /var/lock/sm/17369039-53e9-5c0f-9e27-141a74b1a50d/sr > > This is the state of the log a just few seconds after the command is > run, and there are no more log lines added after this. The log sits > like this during most of the command''s run, and even after the command > finishes with an error. > > > The relevant objects: > > VM (dom0): > [root@localhost ~]# xe vm-list > uuid ( RO) : dd6681cb-f886-4847-b267-a24f555e14ee > name-label ( RW): Control domain on host: xcp > power-state ( RO): running > > > VBD: > [root@localhost ~]# xe vbd-list params=all > uuid ( RO) : 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea > vm-uuid ( RO): dd6681cb-f886-4847-b267-a24f555e14ee > vm-name-label ( RO): Control domain on host: xcp > vdi-uuid ( RO): 2463c214-05a9-49d4-8cba-b60ceb6b16db > vdi-name-label ( RO): testvdi > allowed-operations (SRO): pause; unpause; plug; attach > current-operations (SRO): > empty ( RO): false > device ( RO): xvdb > userdevice ( RW): xvdb > bootable ( RW): false > mode ( RW): RW > type ( RW): Disk > unpluggable ( RW): true > currently-attached ( RO): false > attachable ( RO): <expensive field> > storage-lock ( RO): false > status-code ( RO): 0 > status-detail ( RO): > qos_algorithm_type ( RW): > qos_algorithm_params (MRW): > qos_supported_algorithms (SRO): > other-config (MRW): > io_read_kbs ( RO): <expensive field> > io_write_kbs ( RO): <expensive field> > > > VDI: > [root@localhost ~]# xe vdi-list > uuid=2463c214-05a9-49d4-8cba-b60ceb6b16db params=all > uuid ( RO) : 2463c214-05a9-49d4-8cba-b60ceb6b16db > name-label ( RW): testvdi > name-description ( RW): > is-a-snapshot ( RO): false > snapshot-of ( RO): <not in database> > snapshots ( RO): > snapshot-time ( RO): 19700101T00:00:00Z > allowed-operations (SRO): resize_online; update; resize; > destroy; clone; copy; snapshot > current-operations (SRO): > sr-uuid ( RO): 17369039-53e9-5c0f-9e27-141a74b1a50d > sr-name-label ( RO): Local File SR > vbd-uuids (SRO): 66b1f602-fe0b-f4f7-ba41-47c7f7e01dea > crashdump-uuids (SRO): > virtual-size ( RO): 1073741824 > physical-utilisation ( RO): 5120 > location ( RO): 2463c214-05a9-49d4-8cba-b60ceb6b16db > type ( RO): User > sharable ( RO): false > read-only ( RO): false > storage-lock ( RO): false > managed ( RO): true > parent ( RO): <not in database> > missing ( RO): false > other-config (MRW): > xenstore-data (MRO): scsi/0x12/0x83: > AIMAMQIBAC1YRU5TUkMgIDI0NjNjMjE0LTA1YTktNDlkNC04Y2JhLWI2MGNlYjZiMTZkYiA=; > scsi/0x12/0x80: AIAAEjI0NjNjMjE0LTA1YTktNDkgIA==; vdi-uuid: > 2463c214-05a9-49d4-8cba-b60ceb6b16db; storage-type: ext > sm-config (MRO): > host_OpaqueRef:e41042ed-25cb-1a7c-cbe9-56a110e2440a: RW > on-boot ( RW): persist > allow-caching ( RW): false > tags (SRW): > > > SR: > [root@localhost ~]# xe sr-list > uuid=17369039-53e9-5c0f-9e27-141a74b1a50d params=all > uuid ( RO) : 17369039-53e9-5c0f-9e27-141a74b1a50d > name-label ( RW): Local File SR > name-description ( RW): > host ( RO): xcp > allowed-operations (SRO): forget; VDI.create; VDI.snapshot; > plug; update; destroy; VDI.destroy; scan; VDI.clone; VDI.resize; > unplug > current-operations (SRO): > VDIs (SRO): 2463c214-05a9-49d4-8cba-b60ceb6b16db > PBDs (SRO): d01bc58b-ad8d-3986-b5ba-7a75cad0d89e > virtual-allocation ( RO): 1073741824 > physical-utilisation ( RO): 196935680 > physical-size ( RO): 97221701632 > type ( RO): ext > content-type ( RO): local SR > shared ( RW): false > other-config (MRW): > sm-config (MRO): devserial: > blobs ( RO): > local-cache-enabled ( RO): false > tags (SRW): > > > PBD: > [root@localhost ~]# xe pbd-list > uuid=d01bc58b-ad8d-3986-b5ba-7a75cad0d89e params=all > uuid ( RO) : d01bc58b-ad8d-3986-b5ba-7a75cad0d89e > host ( RO) [DEPRECATED]: 8f1f205a-d352-48cf-afa1-7f2fe1f0f7cf > host-uuid ( RO): 8f1f205a-d352-48cf-afa1-7f2fe1f0f7cf > sr-uuid ( RO): 17369039-53e9-5c0f-9e27-141a74b1a50d > device-config (MRO): device: /dev/hda3 > currently-attached ( RO): true > other-config (MRW): > > TAP-CTL LIST OUTPUT: > 4010 0 0 vhd > /var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d/2463c214-05a9-49d4-8cba-b60ceb6b16db.vhd > > > MOUNTED FS in dom0: > [root@localhost ~]# mount > /dev/hda1 on / type ext3 (rw) > none on /proc type proc (rw) > none on /sys type sysfs (rw) > none on /dev/pts type devpts (rw) > none on /dev/shm type tmpfs (rw) > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > /dev/mapper/XSLocalEXT--17369039--53e9--5c0f--9e27--141a74b1a50d-17369039--53e9--5c0f--9e27--141a74b1a50d > on /var/run/sr-mount/17369039-53e9-5c0f-9e27-141a74b1a50d type ext3 > (rw) >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users