Hi dear list, I''m using xen 4.1.1 (built from tar.gz) with 2.6.39.2 kernel + blkback module and I''m getting into an issue trying to detach a block device from a domain. What I did: root@mybox:~# vhd-util create -s 50000 -n /cloud/data2/machines/vm1.vhd root@mybox:~# xl block-attach 0 tap:vhd:/cloud/local/vm1.vhd xvda rw 0 everything fine, xvda shows up in the dom0 and I can access it. Now i want to detach it to reattach it to another domain: root@mybox:~# xl block-list 0 Vdev BE handle state evt-ch ring-ref BE-path 51728 0 0 4 99 8 /local/domain/0/backend/vbd/0/51728 root@mybox:~# xl block-detach 0 51728 Error: Device 51728 not connected Usage: xm block-detach <Domain> <DevId> [-f|--force] Destroy a domain''s virtual block device. game over :( Any idea what could be wrong ? Bug ? Thanks in advance for your advices. -- Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09.07.2011 16:34, Sébastien Riccio wrote:> Hi dear list, > > I''m using xen 4.1.1 (built from tar.gz) with 2.6.39.2 kernel + blkback > module and I''m getting into an issue trying to detach a block device > from a domain. > > What I did: > > root@mybox:~# vhd-util create -s 50000 -n /cloud/data2/machines/vm1.vhd > root@mybox:~# xl block-attach 0 tap:vhd:/cloud/local/vm1.vhd xvda rw 0 > > everything fine, xvda shows up in the dom0 and I can access it. > > Now i want to detach it to reattach it to another domain: > > root@mybox:~# xl block-list 0 > Vdev BE handle state evt-ch ring-ref BE-path > 51728 0 0 4 99 8 > /local/domain/0/backend/vbd/0/51728 > > root@mybox:~# xl block-detach 0 51728 > Error: Device 51728 not connected > Usage: xm block-detach <Domain> <DevId> [-f|--force] > > Destroy a domain''s virtual block device. > > game over :( > > Any idea what could be wrong ? Bug ? > > Thanks in advance for your advices. > > -- > Sébastien > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >Additions to my previous post: As I can''t detach the block device, the only way (i found) to get rid of it is to reboot the xen host, but it''s not acceptable for a near production system. I''ve tried other ways to do what I wished to do (mount a vhd file as a block device in dom0 in order to be able to fdisk, format it and inject files in it. Try no 2) The readme of blktap2 that is shipped with xen 4.1.1 tar.gz says: -----8<--------------------------------------8<--------------------------------- Mounting images in Dom0 using the blktap2 driver ==============================================Tap (and blkback) disks are also mountable in Dom0 without requiring an active VM to attach. The syntax is - tapdisk2 -n <type>:<full path to file> For example - tapdisk2 -n aio:/home/images/rawFile.img When successful the location of the new device will be provided by tapdisk2 to stdout and tapdisk2 will terminate. From that point forward control of the device is provided through sysfs in the directory- /sys/class/blktap2/blktap#/ Where # is a blktap2 device number present in the path that tapdisk2 printed before terminating. The sysfs interface is largely intuitive, for example, to remove tap device 0 one would- echo 1 > /sys/class/blktap2/blktap0/remove -----8<--------------------------------------8<--------------------------------- So let''s try: mybox# tapdisk2 -n vhd:/cloud/data2/machines/vm1.vhd tapdisk2: invalid option -- ''n'' usage: tapdisk2 <-u uuid> <-c control socket> Huh ? Seems it''s no way either that path. Is it a bug or the README is an outdated one ? I''ve googled around trying to find a solution and it found out about tap-ctl, with examples i found here: http://permalink.gmane.org/gmane.comp.emulators.xen.devel/84223 Okay it''s not as easy as a one-line command that the README shows about tapdisk2, but I have a find a way, so let''s go :) mybox# tap-ctl allocate mybox# tap-ctl spawn mybox# tap-ctl list mybox# tap-ctl attach -p 2546 -m 0 mybox# tap-ctl list mybox# tap-ctl open -p 2546 -m -o -a vhd:/cloud/data2/machines/vm1.vhd Ok seems I can now fdisk the device mybox# fdisk /dev/xen/blktap-2/tapdev0 I''ve setup partition 1 and partition 2 with the scheme I wanted, wrote the changes to it. Results: Disk /dev/xen/blktap-2/tapdev0: 52.4 GB, 52428800000 bytes 255 heads, 63 sectors/track, 6374 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x440cbefb Device Boot Start End Blocks Id System /dev/xen/blktap-2/tapdev0p1 1 6267 50339646 83 Linux /dev/xen/blktap-2/tapdev0p2 6268 6374 859477+ 82 Linux swap / Solaris Good. Now let''s format /dev/xen/blktap-2/tapdev0p1 and mount it so I can inject the files in the partition: mybox# mkfs.ext3 /dev/xen/blktap-2/tapdev0p1 mke2fs 1.41.12 (17-May-2010) Could not stat /dev/xen/blktap-2/tapdev0p1 --- No such file or directory The device apparently does not exist; did you specify it correctly? Hmmm, seems that /dev/xen/blktap-2/tapdev0p1 does not exists. mybox# ls -la /dev/xen/blktap-2/ total 0 drwxr-xr-x 2 root root 100 Jul 9 17:32 . drwxr-xr-x 3 root root 80 Jul 9 17:17 .. crw------- 1 root root 252, 0 Jul 9 17:32 blktap0 crw------- 1 root root 10, 62 Jul 9 17:17 control brw------- 1 root root 254, 0 Jul 9 18:19 tapdev0 Indeed. Does not exists. Now I remember fdisk told be that it wasn''t able to make the kernel to be aware of the changes until I reboot or I use partprobe or kpartx. Let''s go: mybox# partprobe /dev/xen/blktap-2/tapdev0 Error: Error informing the kernel about modifications to partition /dev/xen/blktap-2/tapdev0p1 -- Invalid argument. This means Linux won''t know about any changes you made to /dev/xen/blktap-2/tapdev0p1 until you reboot -- so you shouldn''t mount it or use it in any way before rebooting. Error: Failed to add partition 1 (Invalid argument) hmm nope, let''s go with partx : mybox# partx -a /dev/xen/blktap-2/tapdev0 BLKPG: Invalid argument error adding partition 1 BLKPG: Invalid argument error adding partition 2 BLKPG: Invalid argument error adding partition 3 BLKPG: Invalid argument error adding partition 4 Neither with partx and kpartx. I''m stuck again :( Of course it''s not an acceptable solution if I have to reboot the whole xen host in order to be able to access the partitions. (and I didn''t even tried if it would work, as the main goal was to get around the xm block-detach problem I had at first). I''ve no more ideas about what to try, any ideas, hints, advices ? Thanks a lot, Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2011-07-09 at 13:28 -0400, Sébastien Riccio wrote:> On 09.07.2011 16:34, Sébastien Riccio wrote: > > Hi dear list, > > > > I''m using xen 4.1.1 (built from tar.gz) with 2.6.39.2 kernel + blkback > > module and I''m getting into an issue trying to detach a block device > > from a domain. > > > > What I did: > > > > root@mybox:~# vhd-util create -s 50000 -n /cloud/data2/machines/vm1.vhd > > root@mybox:~# xl block-attach 0 tap:vhd:/cloud/local/vm1.vhd xvda rw 0 > > > > everything fine, xvda shows up in the dom0 and I can access it. > > > > Now i want to detach it to reattach it to another domain: > > > > root@mybox:~# xl block-list 0 > > Vdev BE handle state evt-ch ring-ref BE-path > > 51728 0 0 4 99 8 > > /local/domain/0/backend/vbd/0/51728 > > > > root@mybox:~# xl block-detach 0 51728 > > Error: Device 51728 not connected > > Usage: xm block-detach <Domain> <DevId> [-f|--force] > > > > Destroy a domain''s virtual block device. > > > > game over :( > > > > Any idea what could be wrong ? Bug ? > > > > Thanks in advance for your advices. > > > > -- > > Sébastien > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > Additions to my previous post: > > As I can''t detach the block device, the only way (i found) to get rid of > it is > to reboot the xen host, but it''s not acceptable for a near production > system. > > I''ve tried other ways to do what I wished to do (mount a vhd file as a block > device in dom0 in order to be able to fdisk, format it and inject files > in it. > > Try no 2) > > The readme of blktap2 that is shipped with xen 4.1.1 tar.gz says: > > -----8<--------------------------------------8<--------------------------------- > Mounting images in Dom0 using the blktap2 driver > ==============================================> Tap (and blkback) disks are also mountable in Dom0 without requiring an > active VM to attach. > > The syntax is - > tapdisk2 -n <type>:<full path to file> > > For example - > tapdisk2 -n aio:/home/images/rawFile.img > > When successful the location of the new device will be provided by > tapdisk2 to stdout and tapdisk2 will terminate. From that point > forward control of the device is provided through sysfs in the > directory- > > /sys/class/blktap2/blktap#/ > > Where # is a blktap2 device number present in the path that tapdisk2 > printed before terminating. The sysfs interface is largely intuitive, > for example, to remove tap device 0 one would- > > echo 1 > /sys/class/blktap2/blktap0/remove > -----8<--------------------------------------8<--------------------------------- > > So let''s try: > > mybox# tapdisk2 -n vhd:/cloud/data2/machines/vm1.vhd > tapdisk2: invalid option -- ''n'' > usage: tapdisk2 <-u uuid> <-c control socket> > > Huh ? Seems it''s no way either that path. Is it a bug or the README > is an outdated one ?Completely outdated.> I''ve googled around trying to find a solution and it found out about > tap-ctl, > with examples i found here: > > http://permalink.gmane.org/gmane.comp.emulators.xen.devel/84223 > > Okay it''s not as easy as a one-line command that the README shows about > tapdisk2, > but I have a find a way, so let''s go :) > > mybox# tap-ctl allocate > mybox# tap-ctl spawn > mybox# tap-ctl list > mybox# tap-ctl attach -p 2546 -m 0 > mybox# tap-ctl list > mybox# tap-ctl open -p 2546 -m -o -a vhd:/cloud/data2/machines/vm1.vhdYou can shortcut this as #tap-ctl create -a vhd:/.... and tap-ctl destroy -m <minor> -p <pid> (Say, didn''t we discuss this last week?)> Ok seems I can now fdisk the device > > mybox# fdisk /dev/xen/blktap-2/tapdev0 > > I''ve setup partition 1 and partition 2 with the scheme I wanted, wrote the > changes to it. Results: > > Disk /dev/xen/blktap-2/tapdev0: 52.4 GB, 52428800000 bytes > 255 heads, 63 sectors/track, 6374 cylinders > Units = cylinders of 16065 * 512 = 8225280 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x440cbefb > > Device Boot Start End Blocks > Id System > /dev/xen/blktap-2/tapdev0p1 1 6267 50339646 > 83 Linux > /dev/xen/blktap-2/tapdev0p2 6268 6374 859477+ > 82 Linux swap / Solaris > > Good. > > Now let''s format /dev/xen/blktap-2/tapdev0p1 and mount it so I can > inject the files in the partition: > > mybox# mkfs.ext3 /dev/xen/blktap-2/tapdev0p1 > mke2fs 1.41.12 (17-May-2010) > Could not stat /dev/xen/blktap-2/tapdev0p1 --- No such file or directory > > The device apparently does not exist; did you specify it correctly?It doesn''t indeed. I kept asking a couple people some time ago. Nobody wanted it, because normally the guests partition. We assign a single minor number to each tapdev, with no stride for partitions, so they won''t get created automagically. The fdisk call above just tells you what it thinks those would be called. Use kpartx instead. If you have serious concerns about that, featurewise, now would be a good time to declare those. I might find myself convinced quite easily.> Hmmm, seems that /dev/xen/blktap-2/tapdev0p1 does not exists. > > mybox# ls -la /dev/xen/blktap-2/ > total 0 > drwxr-xr-x 2 root root 100 Jul 9 17:32 . > drwxr-xr-x 3 root root 80 Jul 9 17:17 .. > crw------- 1 root root 252, 0 Jul 9 17:32 blktap0 > crw------- 1 root root 10, 62 Jul 9 17:17 control > brw------- 1 root root 254, 0 Jul 9 18:19 tapdev0 > > Indeed. Does not exists. Now I remember fdisk told be that it wasn''t able to > make the kernel to be aware of the changes until I reboot or I use partprobe > or kpartx.This would rather the case if you reformat mounted disks. Not the case here. If we added partition support, they''d show up (as /dev/tda1, /dev/tda2, ...)> Let''s go: > > mybox# partprobe /dev/xen/blktap-2/tapdev0 > Error: Error informing the kernel about modifications to partition > /dev/xen/blktap-2/tapdev0p1 -- Invalid argument. This means Linux won''t > know about any changes you made to /dev/xen/blktap-2/tapdev0p1 until you > reboot -- so you shouldn''t mount it or use it in any way before rebooting. > Error: Failed to add partition 1 (Invalid argument) > > hmm nope, let''s go with partx : > > mybox# partx -a /dev/xen/blktap-2/tapdev0 > BLKPG: Invalid argument > error adding partition 1 > BLKPG: Invalid argument > error adding partition 2 > BLKPG: Invalid argument > error adding partition 3 > BLKPG: Invalid argument > error adding partition 4Oh, interesting. This one is new to me... I''ll check it out.> Neither with partx and kpartx. I''m stuck again :( > > Of course it''s not an acceptable solution if I have to reboot the whole > xen host in order to be > able to access the partitions.It would not even help.> (and I didn''t even tried if it would > work, as the main goal was > to get around the xm block-detach problem I had at first). > > I''ve no more ideas about what to try, any ideas, hints, advices ?To just-get-me-that partition: Plug into dom0, so you''ll get xvda1-n. Oh wait, that was what didn''t work either in the original mail.. :-) Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10.07.2011 11:05, Daniel Stodden wrote:> To just-get-me-that partition: Plug into dom0, so you''ll get xvda1-n. > Oh wait, that was what didn''t work either in the original mail.. :-) > > Daniel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >Hi Daniel, Thanks a lot for your answers :) Well yes my original problem is to be able to attach and detach a vhd file as a block device in the dom0, so I can provision it. Attaching it to dom0 with block-attach works well, can access, partition it, format partitions, mount partitions, copy files... but then I can''t detach it and I''m stuck :) That''s why I tried other ways to do it and got then stuck again on the "can''t access a partition" problem with the tap-ctl method. So talking about tap-ctl, yes IMHO it would be interesting to be able to access them for such operations. Also I don''t know if my xm/xl block-detach is a bug or happens only on my systems, or maybe only with VHD based disks. If it''s the case, would also be nice to have it fixed. -- Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, 2011-07-10 at 08:07 -0400, Sébastien Riccio wrote:> On 10.07.2011 11:05, Daniel Stodden wrote: > > To just-get-me-that partition: Plug into dom0, so you''ll get xvda1-n. > > Oh wait, that was what didn''t work either in the original mail.. :-) > > > > Daniel > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > Hi Daniel, > > Thanks a lot for your answers :) Well yes my original problem is to be > able to > attach and detach a vhd file as a block device in the dom0, so I can > provision it. > > Attaching it to dom0 with block-attach works well, can access, partition > it, format > partitions, mount partitions, copy files... but then I can''t detach it > and I''m stuck :)Okay, that needs to get fixed, but I don''t know where. In XCP that''s how it''s exclusively done, because it''s the most general approach.> That''s why I tried other ways to do it and got then stuck again on the > "can''t access > a partition" problem with the tap-ctl method. > > So talking about tap-ctl, yes IMHO it would be interesting to be able to > access them > for such operations.Partitions normally work by adding some stride between the ''part0'' (the entire disk) nodes allocated. E.g. scsi disks would allocate every 16th minor for a new disk, and the space in between is partitions. brw-rw---- 1 root disk 8, 0 Jul 7 09:47 /dev/sda brw-rw---- 1 root disk 8, 1 Jul 7 09:47 /dev/sda1 brw-rw---- 1 root disk 8, 2 Jul 7 09:47 /dev/sda2 We don''t have that, so partitions are off. # l /dev/td* brw-rw---- 1 root disk 254, 0 Jul 10 10:58 /dev/tda brw-rw---- 1 root disk 254, 1 Jul 10 11:18 /dev/tdb Problem is I''m not sure if just doing so wouldn''t break sth, because we work so much with minor numbers in the toolstack. kpartx should work, I don''t think that is tapdisk-related. After all it doesn''t really do any magic. It reads the partition table from the start of the device and maybe lands a couple standard ioctls to query disk size etc. The rest is DM. Can you check if it works with some normal disk? Check out modules, install lvm2, make sure you have dm-linear loaded, etc... You were running a custom kernel, right? You''re probably just missing sth. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10.07.2011 20:19, Daniel Stodden wrote:> Okay, that needs to get fixed, but I don''t know where. In XCP that''s > how it''s exclusively done, because it''s the most general approach.My guess at the moment is that it might be a problem with blktap and vhd or, blktap and my kernel combo, or blktap and my kernel combo and vhd :) The kernel i''m actually playing with is: 2.6.39.2-xen-stable + blktap, built over a debian squeeze like this: git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git linux_xen_2.6.39.x-stable cd linux_xen_2.6.39.x-stable git checkout -b stable/2.6.39.x origin/stable/2.6.39.x git remote add daniel git://xenbits.xensource.com/people/dstodden/linux.git git fetch daniel git merge daniel/blktap/next-2.6.39 make menuconfig (removing useless stuff, activating the needed ones) make bzImage -j9 ; make modules -j9 ; make modules_install [ etc ... ] Playing more with it this morning I managed to start a vm with a vhd file for the disk (after provisioning it with some files and reboots...) , with this config: box# cat /cloud/data2/configs/vm1.test.cfg bootloader = "/usr/bin/pygrub" memory = 1024 name = "vm1" vcpus = 4 #vif = [ ''ip=10.111.5.10, bridge=trunk0, vifname=vm1.0'' ] disk = [ ''tap2:vhd:/cloud/data2/machines/vm1.vhd,xvda,w'' ] root = "/dev/xvda1" extra = "fastboot" on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' vm1 is up and rocking box# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1024 16 r----- 26.2 vm1 2 1024 4 -b---- 2.8 But now if I issue a ps -aux in the dom0, it displays some process then hangs the ps. (that was not the case before I start vm1) And if I try to list the attached block devices with xl: box# xl block-list 2 Vdev BE handle state evt-ch ring-ref BE-path Segmentation fault (dmesg) [ 1592.151122] xl[2292]: segfault at 0 ip 00007f7de314e6d2 sp 00007fff610e30b0 error 4 in libc-2.11.2.so[7f7de3117000+158000] but works if i try to list it with xm: box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 I''ll try with something else than vhd to see if the same happens, but my goal is to use vhd''s ...> Can you check if it works with some normal disk? Check out modules, > install lvm2, make sure you have dm-linear loaded, etc... You were > running a custom kernel, right? You''re probably just missing sth.modules lists: root@xen-blade15:~# lsmod Module Size Used by blktap 17941 8 ocfs2 618206 1 quota_tree 7539 1 ocfs2 ocfs2_dlmfs 17331 1 ocfs2_stack_o2cb 3482 1 ocfs2_dlm 204671 1 ocfs2_stack_o2cb ocfs2_nodemanager 186569 14 ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm ocfs2_stackglue 7437 3 ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb dm_round_robin 2260 1 configfs 21658 2 ocfs2_nodemanager crc32c 2688 8 iscsi_tcp 8503 6 libiscsi_tcp 11604 1 iscsi_tcp libiscsi 34844 2 iscsi_tcp,libiscsi_tcp scsi_transport_iscsi 28673 3 iscsi_tcp,libiscsi openvswitch_mod 71205 3 xenfs 9815 1 xfs 501098 1 ext2 61369 1 sg 27333 0 sr_mod 14760 0 cdrom 35494 1 sr_mod xen_evtchn 4739 2 loop 16002 0 tpm_tis 7821 0 tpm 10878 1 tpm_tis i7core_edac 15891 0 tpm_bios 4921 1 tpm dcdbas 5416 0 edac_core 34483 1 i7core_edac evdev 9374 4 usb_storage 43361 0 thermal_sys 14045 0 pcspkr 1779 0 acpi_processor 5423 0 [permanent] button 4199 0 usbhid 34740 0 hid 78436 1 usbhid ext4 255423 1 mbcache 5434 2 ext2,ext4 jbd2 48549 2 ocfs2,ext4 crc16 1319 1 ext4 dm_multipath 16384 2 dm_round_robin scsi_dh 4876 1 dm_multipath dm_mod 63657 7 dm_multipath sd_mod 34293 6 crc_t10dif 1292 1 sd_mod uhci_hcd 21828 0 megaraid_sas 70747 3 ehci_hcd 37665 0 scsi_mod 144719 9 iscsi_tcp,libiscsi,scsi_transport_iscsi,sg,sr_mod,usb_storage,scsi_dh,sd_mod,megaraid_sas usbcore 137744 5 usb_storage,usbhid,uhci_hcd,ehci_hcd bnx2 70964 0 My vhd storage is on an ocfs2 shared storage attached with multipath iscsi. I will try it on a local storage too to eliminate that possible cause. Thanks for your time Daniel. Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2011-07-11 at 13:09 -0400, Sébastien Riccio wrote:> On 10.07.2011 20:19, Daniel Stodden wrote: > > Okay, that needs to get fixed, but I don''t know where. In XCP that''s > > how it''s exclusively done, because it''s the most general approach. > > My guess at the moment is that it might be a problem with blktap and > vhd or, blktap and my kernel combo, or blktap and my kernel combo and > vhd :) > > The kernel i''m actually playing with is: 2.6.39.2-xen-stable + blktap, > built over a debian squeeze like this: > > git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git > linux_xen_2.6.39.x-stable > cd linux_xen_2.6.39.x-stable > git checkout -b stable/2.6.39.x origin/stable/2.6.39.x > git remote add daniel > git://xenbits.xensource.com/people/dstodden/linux.git > git fetch daniel > git merge daniel/blktap/next-2.6.39 > make menuconfig (removing useless stuff, activating the needed ones) > make bzImage -j9 ; make modules -j9 ; make modules_install > [ etc ... ]I''m not familiar with potential open issues in konrad''s 2.6.39 tree, but I wouldn''t expect that to be the problem.> Playing more with it this morning I managed to start a vm with a vhd > file for the disk (after provisioning it with some files and > reboots...) , with this config: > > box# cat /cloud/data2/configs/vm1.test.cfg > bootloader = "/usr/bin/pygrub" > memory = 1024 > name = "vm1" > vcpus = 4 > #vif = [ ''ip=10.111.5.10, bridge=trunk0, vifname=vm1.0'' ] > disk = [ ''tap2:vhd:/cloud/data2/machines/vm1.vhd,xvda,w'' ] > root = "/dev/xvda1" > extra = "fastboot" > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart''Looks good, although I''m not really good with the ''disk'' lines either. E.g. mine just say tap:aio, tap:vhd, etc.> > vm1 is up and rocking > > box# xl list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 1024 16 r----- > 26.2 > vm1 2 1024 4 -b---- > 2.8 > > But now if I issue a ps -aux in the dom0, it displays some process > then hangs the ps. > (that was not the case before I start vm1) > > And if I try to list the attached block devices with xl: > > box# xl block-list 2 > Vdev BE handle state evt-ch ring-ref BE-path > Segmentation faultOuch. What xen tree are you running? Unstable? What does tap-ctl list say, does that work? You might want to try the 4.1 tree instead. If not, you''ll at least want to get yourself a coredump to get an idea of what you''re after.> (dmesg) > [ 1592.151122] xl[2292]: segfault at 0 ip 00007f7de314e6d2 sp > 00007fff610e30b0 error 4 in libc-2.11.2.so[7f7de3117000+158000]# ulimit -c unlimited # xl block-list 2 ... core dumped. # gdb $(which xl) core (gdb) backtrace> but works if i try to list it with xm: > > box# xm block-list 2 > Vdev BE handle state evt-ch ring-ref BE-path > 51712 0 0 4 23 > 8 /local/domain/0/backend/vbd/2/51712Well, different codebase...> > I''ll try with something else than vhd to see if the same happens, but > my goal is to use vhd''s ...Just check if the tapdisk''s are all working. You didn''t see those failing. But if your guests look happy (xl console, poke around) long as, your problem is elsewhere.> > > > Can you check if it works with some normal disk? Check out modules, > > install lvm2, make sure you have dm-linear loaded, etc... You were > > running a custom kernel, right? You''re probably just missing sth. > > modules lists: > > root@xen-blade15:~# lsmod > Module Size Used by > blktap 17941 8 > ocfs2 618206 1 > quota_tree 7539 1 ocfs2 > ocfs2_dlmfs 17331 1 > ocfs2_stack_o2cb 3482 1 > ocfs2_dlm 204671 1 ocfs2_stack_o2cb > ocfs2_nodemanager 186569 14 > ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm > ocfs2_stackglue 7437 3 ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb > dm_round_robin 2260 1 > configfs 21658 2 ocfs2_nodemanager > crc32c 2688 8 > iscsi_tcp 8503 6 > libiscsi_tcp 11604 1 iscsi_tcp > libiscsi 34844 2 iscsi_tcp,libiscsi_tcp > scsi_transport_iscsi 28673 3 iscsi_tcp,libiscsi > openvswitch_mod 71205 3 > xenfs 9815 1 > xfs 501098 1 > ext2 61369 1 > sg 27333 0 > sr_mod 14760 0 > cdrom 35494 1 sr_mod > xen_evtchn 4739 2 > loop 16002 0 > tpm_tis 7821 0 > tpm 10878 1 tpm_tis > i7core_edac 15891 0 > tpm_bios 4921 1 tpm > dcdbas 5416 0 > edac_core 34483 1 i7core_edac > evdev 9374 4 > usb_storage 43361 0 > thermal_sys 14045 0 > pcspkr 1779 0 > acpi_processor 5423 0 [permanent] > button 4199 0 > usbhid 34740 0 > hid 78436 1 usbhid > ext4 255423 1 > mbcache 5434 2 ext2,ext4 > jbd2 48549 2 ocfs2,ext4 > crc16 1319 1 ext4 > dm_multipath 16384 2 dm_round_robin > scsi_dh 4876 1 dm_multipath > dm_mod 63657 7 dm_multipathDo you have dm-linear (I think it''s just linear.ko) available? If not, it might explain yesterday''s kpartx issue.> sd_mod 34293 6 > crc_t10dif 1292 1 sd_mod > uhci_hcd 21828 0 > megaraid_sas 70747 3 > ehci_hcd 37665 0 > scsi_mod 144719 9 > iscsi_tcp,libiscsi,scsi_transport_iscsi,sg,sr_mod,usb_storage,scsi_dh,sd_mod,megaraid_sas > usbcore 137744 5 usb_storage,usbhid,uhci_hcd,ehci_hcd > bnx2 70964 0 > > My vhd storage is on an ocfs2 shared storage attached with multipath > iscsi. I will try it on a local storage too to eliminate > that possible cause.Well, yeah, that''s a bit thicker than normally recommended for testing patchworks, but then again, it doesn''t really sound like that''s your most immediate problem. Cheers, Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11.07.2011 20:22, Daniel Stodden wrote:> Looks good, although I''m not really good with the ''disk'' lines either. > E.g. mine just say tap:aio, tap:vhd, etc.Hmm what is the difference betwwen tap:vhd and tap2:vhd ? If I start my vm1 using tap:vhd it does not hangs on a "ps -aux" in the dom0, and the xl block-list 0 doesn''t segfault, but it does if I use tap2:vhd. I''m a bit confused. There is blktap and blktap2 in the xen sources, but one module blktap for the kernel. The same module is used for both versions ?> Ouch. What xen tree are you running? Unstable? What does tap-ctl list > say, does that work? You might want to try the 4.1 tree instead. If > not, you''ll at least want to get yourself a coredump to get an idea of > what you''re after.I''m running 4.1.1 from the tar.gz, also tried 4.1.0, same things occurs. I''m still investigating :) Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2011-07-11 at 17:37 -0400, Sébastien Riccio wrote:> On 11.07.2011 20:22, Daniel Stodden wrote: > > > Looks good, although I''m not really good with the ''disk'' lines either. > > E.g. mine just say tap:aio, tap:vhd, etc. > > Hmm what is the difference betwwen tap:vhd and tap2:vhd ?There''s a lot of compat hell in there. If either one or the other works for you, you''ll probably regret you asked. :o)> If I start my vm1 using tap:vhd it does not hangs on a "ps -aux" in the > dom0, and the xl block-list 0 doesn''t segfault, but it does if I use > tap2:vhd.Ah. Interesting. That might indicate a simple problem with a simple solution.> I''m a bit confused. There is blktap and blktap2 in the xen sources, but > one module blktap for the kernel. The same module is used for both > versions ?There used to be a blktap1. It''s quite different, and supposed to die. It only lived in 2.6.18. Don''t bother.> > Ouch. What xen tree are you running? Unstable? What does tap-ctl list > > say, does that work? You might want to try the 4.1 tree instead. If > > not, you''ll at least want to get yourself a coredump to get an idea of > > what you''re after. > > I''m running 4.1.1 from the tar.gz, also tried 4.1.0, same things occurs. > I''m still investigating :)Okay, very cool. If you even happen to come across a fix for the tap2 issue, or at least identify the crash cause more precisely, that''d be great. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi again :) Going further with my investigations, I''m definitively now sure that it''s not an issue with ocfs2. Same behavior occures on a local file. There is really something going wrong with xm/xl. In order to continue my tests I''ve tried to attach and .img (raw) file and this time not attaching it to dom0 but to a running domU vm: no way with xl box# xl block-attach 2 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20110428.img xvdc w libxl_device_disk_add failed. with xm box# xm block-attach 2 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20110428.img xvdc w this worked. Now trying to list with xl : box# xl block-list 2 Vdev BE handle state evt-ch ring-ref BE-path Segmentation fault box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 *51744 0 0 4 25 50 /local/domain/0/backend/vbd/2/51744 * Unplugging with xm: box# xm block-detach 2 51744 box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 Gone :) Summary: 1) block-attach / block-detach / block-list actions issued from xl and xm are not having the same behaviors, xl tend to crash a lot and mess things. 2) attaching an .img file with xm and file:/ let me detach it with xm block-detach 3) attaching a .vhd file with xm and tap:vhd:/ does *not* let me detach it with xm block-detach I almost got rid of all specialities on my box, except maybe konrad''s 2.6.39.2 kernel. This is a quite serious issue, am I the only one able to reproduce it ? For information, we''ve tried the same action (xl block-attach and xl block-detach of a vhd on XenServer 6.0 beta (project boston) and the same issue happens, can''t detach it, then it throws ugly messages at console: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: ------------[ cut here ]------------ Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: invalid opcode: 0000 [#1] SMP Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: last sysfs file: /sys/devices/xen-backend/vbd-0-51712/statistics/rd_usecs Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Process xenwatch (pid: 48, ti=ee9d0000 task=ee8d5070 task.ti=ee9d0000) Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Stack: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Call Trace: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Code: 88 ff ff e9 5c ff ff ff 89 44 24 04 c7 44 24 08 2e a1 44 c0 8b 4d ec 89 0c 24 e8 62 88 ff ff 89 f8 e8 db ee ff ff e9 f7 fe ff ff <0f> 0b eb fe 66 90 ba 98 00 00 00 b8 ba a0 44 c0 e8 61 48 e8 ff Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: EIP: [<c02a985a>] blkback_queue_start+0x2ca/0x2f0 SS:ESP 0069:ee9d1f38 So my conclusions are that something is not right with xl/xm commands and block attaching/detaching. I guess XenServer is not using this method to attach/detach disks as attaching them with xe commands and with xencenter do not show any problems. Could someone try it on their side and confirms if this is a confirmed problem ? That would be nice to fix it if it''s the case, as shipping xen with a "russian roulette" enabled xm/xl is maybe not the best :) I can provide root access to my test machines if you want to try things on the best :) Cheers, Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11.07.2011 20:22, Daniel Stodden wrote:> ulimit -c unlimited > # xl block-list 2 > ... core dumped. > # gdb $(which xl) core >Results : Core was generated by `xl block-list 2''. Program terminated with signal 11, Segmentation fault. #0 0x00007fda1ed796d2 in ?? () from /lib/libc.so.6 (gdb) backtrace #0 0x00007fda1ed796d2 in ?? () from /lib/libc.so.6 #1 0x00007fda1ff10e13 in libxl_append_disk_list_of_type () from /usr/lib/libxenlight.so.1.0 #2 0x00007fda1ff10ee2 in libxl_device_disk_list () from /usr/lib/libxenlight.so.1.0 #3 0x000000000040a483 in main_blocklist () #4 0x0000000000405442 in main () _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
More informations coming: Vhd attached with: xl block-attach 2 tap:vhd:/cloud/data2/machines/vm2.vhd xvdb w Strace results of detach attempt with xl block-detach in attached file xl_vhd_detach.txt (seems to receive an OK but it doesn''t detach anything) Strace results of detach attempt with xm block-detach in attached file xm_vhd_detach.txt (doesn''t seem to know anything about the device existence) Hope this can help for debugging the things out :) Cheers, Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi again :) Going further with my investigations, I''m definitively now sure that it''s not an issue with ocfs2. Same behavior occures on a local file. There is really something going wrong with xm/xl. In order to continue my tests I''ve tried to attach and .img (raw) file and this time not attaching it to dom0 but to a running domU vm: no way with xl box# xl block-attach 2 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20110428.img xvdc w libxl_device_disk_add failed. with xm box# xm block-attach 2 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20110428.img xvdc w this worked. Now trying to list with xl : box# xl block-list 2 Vdev BE handle state evt-ch ring-ref BE-path Segmentation fault box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 *51744 0 0 4 25 50 /local/domain/0/backend/vbd/2/51744 * Unplugging with xm: box# xm block-detach 2 51744 box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 Gone :) Summary: 1) block-attach / block-detach / block-list actions issued from xl and xm are not having the same behaviors, xl tend to crash a lot and mess things. 2) attaching an .img file with xm and file:/ let me detach it with xm block-detach 3) attaching a .vhd file with xm and tap:vhd:/ does *not* let me detach it with xm block-detach I almost got rid of all specialities on my box, except maybe konrad''s 2.6.39.2 kernel. This is a quite serious issue, am I the only one able to reproduce it ? For information, we''ve tried the same action (xl block-attach and xl block-detach of a vhd on XenServer 6.0 beta (project boston) and the same issue happens, can''t detach it, then it throws ugly messages at console: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: ------------[ cut here ]------------ Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: invalid opcode: 0000 [#1] SMP Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: last sysfs file: /sys/devices/xen-backend/vbd-0-51712/statistics/rd_usecs Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Process xenwatch (pid: 48, ti=ee9d0000 task=ee8d5070 task.ti=ee9d0000) Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Stack: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Call Trace: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Code: 88 ff ff e9 5c ff ff ff 89 44 24 04 c7 44 24 08 2e a1 44 c0 8b 4d ec 89 0c 24 e8 62 88 ff ff 89 f8 e8 db ee ff ff e9 f7 fe ff ff <0f> 0b eb fe 66 90 ba 98 00 00 00 b8 ba a0 44 c0 e8 61 48 e8 ff Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: EIP: [<c02a985a>] blkback_queue_start+0x2ca/0x2f0 SS:ESP 0069:ee9d1f38 So my conclusions are that something is not right with xl/xm commands and block attaching/detaching. I guess XenServer is not using this method to attach/detach disks as attaching them with xe commands and with xencenter do not show any problems. Could someone try it on their side and confirms if this is a confirmed problem ? That would be nice to fix it if it''s the case, as shipping xen with a "russian roulette" enabled xm/xl is maybe not the best :) I can provide root access to my test machines if you want to try things on the best :) Cheers, Sébastien _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Nobody getting the same issues with xen 4.1.x and xl block-attach/detach of vhd files ? As i''m stuck with that, I''ve tried to give a try with xen-unstable tree. Seems that the syntax for block-attach has changed. Anyone has a clue about how to attach a vhd file as a block device using xl block-attach ? also with 4.2-unstable disk = [ ''tap:vhd:/cloud/data2/machines/vm1.vhd,xvda,w'' ] does not work any more Parsing config file /cloud/data2/configs/vm1.test.cfg libxl: error: libxl.c:1102:libxl_device_disk_local_attach: cannot locally attach a qdisk image if the format is not raw libxl: error: libxl_create.c:437:do_domain_create: failed to run bootloader: -3 Thanks, Sébastien On 12.07.2011 07:44, Sébastien Riccio wrote:> More informations coming: > > Vhd attached with: > > xl block-attach 2 tap:vhd:/cloud/data2/machines/vm2.vhd xvdb w > > Strace results of detach attempt with xl block-detach in attached > file xl_vhd_detach.txt > (seems to receive an OK but it doesn''t detach anything) > > Strace results of detach attempt with xm block-detach in attached file > xm_vhd_detach.txt > (doesn''t seem to know anything about the device existence) > > Hope this can help for debugging the things out :) > > Cheers, > Sébastien >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Nobody getting the same issues with xen 4.1.x and xl block-attach/detach of vhd files ? As i''m stuck with that, I''ve tried to give a try with xen-unstable tree. Seems that the syntax for block-attach has changed. Anyone has a clue about how to attach a vhd file as a block device using xl block-attach ? also with 4.2-unstable disk = [ ''tap:vhd:/cloud/data2/machines/vm1.vhd,xvda,w'' ] does not work any more Parsing config file /cloud/data2/configs/vm1.test.cfg libxl: error: libxl.c:1102:libxl_device_disk_local_attach: cannot locally attach a qdisk image if the format is not raw libxl: error: libxl_create.c:437:do_domain_create: failed to run bootloader: -3 Thanks, Sébastien On 12.07.2011 07:44, Sébastien Riccio wrote:> More informations coming: > > Vhd attached with: > > xl block-attach 2 tap:vhd:/cloud/data2/machines/vm2.vhd xvdb w > > Strace results of detach attempt with xl block-detach in attached > file xl_vhd_detach.txt > (seems to receive an OK but it doesn''t detach anything) > > Strace results of detach attempt with xm block-detach in attached file > xm_vhd_detach.txt > (doesn''t seem to know anything about the device existence) > > Hope this can help for debugging the things out :) > > Cheers, > Sébastien >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jul 13, 2011 at 10:44:27AM +0200, Sébastien Riccio wrote:> Hi, > > Nobody getting the same issues with xen 4.1.x and xl > block-attach/detach of vhd files ?Hey Sebastien, I hadn''t actually used vhd files so I can''t comment on that. There was bug that was found by Joe Jin (http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00275.html) but that is when the disk is mounted - but you don''t even get to that. Is this _only_ happening with vhds? does it appear to exist with phy''s?> > As i''m stuck with that, I''ve tried to give a try with xen-unstable > tree. Seems that > the syntax for block-attach has changed. Anyone has a clue about how > to attach > a vhd file as a block device using xl block-attach ? > > also with 4.2-unstable > disk = [ ''tap:vhd:/cloud/data2/machines/vm1.vhd,xvda,w'' ] > > does not work any more > > Parsing config file /cloud/data2/configs/vm1.test.cfg > libxl: error: libxl.c:1102:libxl_device_disk_local_attach: cannot > locally attach a qdisk image if the format is not raw > libxl: error: libxl_create.c:437:do_domain_create: failed to run > bootloader: -3That is expected .. the upstream version of QEMU does not have all the bells and whistles the older versions had.> > Thanks, > Sébastien > On 12.07.2011 07:44, Sébastien Riccio wrote: > >More informations coming: > > > >Vhd attached with: > > > >xl block-attach 2 tap:vhd:/cloud/data2/machines/vm2.vhd xvdb w > > > >Strace results of detach attempt with xl block-detach in attached > >file xl_vhd_detach.txt > >(seems to receive an OK but it doesn''t detach anything) > > > >Strace results of detach attempt with xm block-detach in attached file > >xm_vhd_detach.txt > >(doesn''t seem to know anything about the device existence) > > > >Hope this can help for debugging the things out :) > > > >Cheers, > >Sébastien > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel