I think I''ve found one reason why we can''t get PV block drivers on HVM domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux domU kernel. We are using a line like: disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] We are using effectively a standard 3.0 kernel. Config options including the word XEN are below. We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package does not contain blktapctrl at all (which would explain why it doesn''t start). Do we need this? It has been suggested that we don''t need this, but we do need a kernel module that provides blktap. http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html suggests these might be queued for the non-existent "2.6.40", but it''s suggested these aren''t in 3.0. http://wiki.xensource.com/xenwiki/XAPI_on_debian suggests there are dmks modules available, but that blktap is currenly only 32 bit. Can that be correct? Isn''t this what blkback does? domU config is below. Note the very same VM with the same config file works just fine on xen 3.3.1. This domU is Centos 2.6.18 with unmodified_drivers xenlinux type kernel (as supplied by Centos). Every other kernel we''ve tried does the same, save that modern ones also unplug the emulated devices so no disks appear as well. PV nics work fine. -- Alex Bligh root@xen4:/home/ubuntu/kernel# fgrep XEN /boot/config-3.0.0-12-server CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=128 CONFIG_XEN_SAVE_RESTORE=y # CONFIG_XEN_DEBUG_FS is not set # CONFIG_XEN_DEBUG is not set CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=m CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m CONFIG_HVC_XEN=y CONFIG_XEN_WDT=m CONFIG_XEN_FBDEV_FRONTEND=m CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XENFS=m CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=m CONFIG_XEN_GNTDEV=m CONFIG_XEN_GRANT_DEV_ALLOC=m CONFIG_XEN_PLATFORM_PCI=m CONFIG_SWIOTLB_XEN=y root@xen4:/home/ubuntu/kernel# lsmod | fgrep -i xen xen_netfront 26671 0 xen_blkfront 26261 0 xen_wdt 13525 0 xen_gntalloc 13321 0 xenbus_probe_frontend 13232 2 xen_netfront,xen_blkfront,[permanent] xen_gntdev 17676 2 netxen_nic 97283 0 xen_netback 27854 0 [permanent] xen_blkback 23177 0 [permanent] xen_evtchn 13172 5 xenfs 18311 1 Guest config looks like this, though we''ve used xvda and hda root@xen4:/home/ubuntu/kernel# cat !$ cat /etc/xen/centos5-pvd name = "centos5-pvd-xen4" uuid = "6d72aff9-8f53-beb7-c71c-89e72b93193e" maxmem = 512 memory = 512 vcpus = 2 builder = "hvm" #kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 1 localtime = 0 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" #device_model = "/usr/lib64/xen/bin/pv-qemu-dm" #device_model = "/usr/lib/xen/bin/stubdom-dm" #sdl = 0 #vnc = 1 vnclisten="0.0.0.0" keymap="en-gb" #vncunused = 1 #vncdisplay = "3" vfb = [ "type=vnc,vncunused=1,keymap=en-gb" ] disk = [ "file:/root/Iain2011/centos-pvd.img,xvda,w" ] vif = [ "mac=00:16:3e:00:a5:58,bridge=br0,type=foo","mac=00:16:3e:00:a5:58,bridge=br0,type=ioemu" ] serial = "pty" _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-27 13:35 UTC
Re: [Xen-devel] PV drivers on HVM using Xen 4.1.1
On Thu, Oct 27, 2011 at 01:56:41PM +0100, Alex Bligh wrote:> I think I''ve found one reason why we can''t get PV block drivers on HVM > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux > domU kernel. > > We are using a line like: > > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > We are using effectively a standard 3.0 kernel. Config options including > the word XEN are below. > > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package > does not contain blktapctrl at all (which would explain why it doesn''t > start). Do we need this? > > It has been suggested that we don''t need this, but we do need a kernel > module that provides blktap. > > http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html > > suggests these might be queued for the non-existent "2.6.40", but > it''s suggested these aren''t in 3.0.Right, they are a no-go. There is a> > http://wiki.xensource.com/xenwiki/XAPI_on_debian > > suggests there are dmks modules available, but that blktap is currenly > only 32 bit. Can that be correct?There is a 64-bit (and 32-bit) version on Daniel''s git tree: git://xenbits.xensource.com/people/dstodden/linux.git .. but the deal is that it is unmaintained (Daniel left Citrix). Thought interestingly .. he made a version of it where all of the fiddling with the generic code has been removed. Neat.> > Isn''t this what blkback does?Blkback can''t handle files - it can only handle block devices.> > domU config is below. Note the very same VM with the same config file > works just fine on xen 3.3.1. This domU is Centos 2.6.18 with > unmodified_drivers xenlinux type kernel (as supplied by Centos). Every > other kernel we''ve tried does the same, save that modern ones also > unplug the emulated devices so no disks appear as well. > > PV nics work fine._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 13:56 +0100, Alex Bligh wrote:> I think I''ve found one reason why we can''t get PV block drivers on HVM > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux > domU kernel. > > We are using a line like: > > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > We are using effectively a standard 3.0 kernel. Config options including > the word XEN are below. > > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > start in Xen3.Are you changing dom0 kernel as well as hypervisor when you say "3" and "4.1.1" or is the dom0 kernel constant?> In fact as far as I can tell the Ubuntu Xen4 package > does not contain blktapctrl at all (which would explain why it doesn''t > start). Do we need this?AFAIK tap:aio: usually requires blktap (both kernel side and userspace). There is also a block backend in qemu which can be by xl used under some circumstances when blktap is not available but the same is not true of xend.> It has been suggested that we don''t need this, but we do need a kernel > module that provides blktap. > > http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html > > suggests these might be queued for the non-existent "2.6.40", but > it''s suggested these aren''t in 3.0. > > http://wiki.xensource.com/xenwiki/XAPI_on_debian > > suggests there are dmks modules available, but that blktap is currenly > only 32 bit. Can that be correct?XCP (from whence that blktap version comes) is only developed/test using 32 bit. Taht''s not to say 64 bit doesn''t work, but it might not.> Isn''t this what blkback does?blkback can only export raw block devices to guests. blktap can be used to export more "structured" image types, such as vhd. A raw img like you have cannot be directly handled by blkback (since it is a file and not a block device). You can consider it a dengenerate case of a structured image so blktap can deal with it. A raw image can also be mounted with a loop back device which itself can be exported via blkback. Some toolstacks (xend) can do this automatically (but won''t for a tap:aio: AFAIK) but xl will not. Unfortunately blktap (at least in its current form) is not suitable to go upstream. If someone wants to step up and implement the blkring protocol in userspace (i.e. do away with the kernel component by merging that bit into the tapdisk process itself) then that would be very much appreciated.> domU config is below. Note the very same VM with the same config file > works just fine on xen 3.3.1. This domU is Centos 2.6.18 with > unmodified_drivers xenlinux type kernel (as supplied by Centos). Every > other kernel we''ve tried does the same, save that modern ones also > unplug the emulated devices so no disks appear as well. > > PV nics work fine. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 13:54 UTC
Re: [Xen-devel] PV drivers on HVM using Xen 4.1.1
On Thu, 27 Oct 2011, Alex Bligh wrote:> I think I''ve found one reason why we can''t get PV block drivers on HVM > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux > domU kernel. > > We are using a line like: > > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ]If this is an HVM guest, you should use hda here because you want to make sure that an emulated IDE disk is created for you as well. Also if /tmp/centos-pvd.img is a raw file, you might as well use file: rather than tap:aio.> We are using effectively a standard 3.0 kernel. Config options including > the word XEN are below. > > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > start in Xen3. In fact as far as I can tell the Ubuntu Xen4 package > does not contain blktapctrl at all (which would explain why it doesn''t > start). Do we need this? > > It has been suggested that we don''t need this, but we do need a kernel > module that provides blktap. > > http://www.vr.org/knowledgebase/1112/Xen-Paravirt-Ops.html > > suggests these might be queued for the non-existent "2.6.40", but > it''s suggested these aren''t in 3.0. > > http://wiki.xensource.com/xenwiki/XAPI_on_debian > > suggests there are dmks modules available, but that blktap is currenly > only 32 bit. Can that be correct? > > Isn''t this what blkback does?blkback only works with block devices. You can still use blkback if you configure a loop device: losetup /dev/loop0 /tmp/centos-pvd.img and then: disk = [ "phy:/dev/loop0,hda,w" ] However is not going to be very fast unfortunately. As an alternative you could download and configure upstream qemu with linux aio and then use it as device model or just block backend but unfortunately it needs some hacking on 4.1 (on 4.2 is semi automated now). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian, --On 27 October 2011 14:44:49 +0100 Ian Campbell <Ian.Campbell@citrix.com> wrote:>> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does >> start in Xen3. > > Are you changing dom0 kernel as well as hypervisor when you say "3" and > "4.1.1" or is the dom0 kernel constant?No, the Xen3 kernel is 2.6.18 Centos, and the Xen 4 kernel is 3.0 Ubuntu Oneiric.>> In fact as far as I can tell the Ubuntu Xen4 package >> does not contain blktapctrl at all (which would explain why it doesn''t >> start). Do we need this? > > AFAIK tap:aio: usually requires blktap (both kernel side and userspace). > There is also a block backend in qemu which can be by xl used under some > circumstances when blktap is not available but the same is not true of > xend.We are using xl (though have tried xend).> XCP (from whence that blktap version comes) is only developed/test using > 32 bit. Taht''s not to say 64 bit doesn''t work, but it might not.OK. The idea here is a production xen4.1 system, so "might not work" sounds like a bad idea :-)>> Isn''t this what blkback does? > > blkback can only export raw block devices to guests.We will in production use raw block devices, so that''s ok. We were just using files for testing. Is the same restriction there is Xen3.3, because it works there (possibly for the reason below).> A raw image can > also be mounted with a loop back device which itself can be exported via > blkback. Some toolstacks (xend) can do this automatically (but won''t for > a tap:aio: AFAIK) but xl will not.I think xl /must/ be doing this, or it is difficult to explain the behaviour below. We discovered Ubuntu''s Xen 4.1 hypervisor package does not contain blktapctrl, so we copied that across from a build from source, plus some libraries, and did an mknod on /dev/xen/blktap0 (which it appeared to want to open). pvdrivers then appeared in the guest. But we then reversed everything we''d done (we think), and pvdrivers continued to appear. We are reinstalling to find out exactly what it was that changed. I must admit to remaining almost totally confused as to how this is meant to work, but thanks for your help thus far! -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:04 +0100, Alex Bligh wrote:> Ian, > > --On 27 October 2011 14:44:49 +0100 Ian Campbell <Ian.Campbell@citrix.com> > wrote: > > > >> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > >> start in Xen3. > > > > Are you changing dom0 kernel as well as hypervisor when you say "3" and > > "4.1.1" or is the dom0 kernel constant? > > No, the Xen3 kernel is 2.6.18 Centos, and the Xen 4 kernel is 3.0 > Ubuntu Oneiric.OK, that is important information now that Xen is not supplied with a particular kernel.> >> In fact as far as I can tell the Ubuntu Xen4 package > >> does not contain blktapctrl at all (which would explain why it doesn''t > >> start). Do we need this? > > > > AFAIK tap:aio: usually requires blktap (both kernel side and userspace). > > There is also a block backend in qemu which can be by xl used under some > > circumstances when blktap is not available but the same is not true of > > xend. > > We are using xl (though have tried xend).Good to know.> > XCP (from whence that blktap version comes) is only developed/test using > > 32 bit. Taht''s not to say 64 bit doesn''t work, but it might not. > > OK. The idea here is a production xen4.1 system, so "might not work" > sounds like a bad idea :-) > > >> Isn''t this what blkback does? > > > > blkback can only export raw block devices to guests. > > We will in production use raw block devices, so that''s ok. We were > just using files for testing. Is the same restriction there is > Xen3.3, because it works there (possibly for the reason below).The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a big factor. blkback.ko is in both so you should be ok. For testing you could try using loopback like Stefano suggests.> > A raw image can > > also be mounted with a loop back device which itself can be exported via > > blkback. Some toolstacks (xend) can do this automatically (but won''t for > > a tap:aio: AFAIK) but xl will not. > > I think xl /must/ be doing this, or it is difficult to explain the > behaviour below.Indeed, I can''t explain it either. I think I would expect xl to fall back to the qemu based backend if blktap wasn''t available. I _think_ that functionality was backported to the qemu-xen in 4.1, but I''m not sure.> We discovered Ubuntu''s Xen 4.1 hypervisor package does not contain > blktapctrl, so we copied that across from a build from source, plus some > libraries, and did an mknod on /dev/xen/blktap0 (which it appeared to want > to open). pvdrivers then appeared in the guest. But we then reversed > everything we''d done (we think), and pvdrivers continued to appear. We are > reinstalling to find out exactly what it was that changed. > > I must admit to remaining almost totally confused as to how this is > meant to work, but thanks for your help thus far! >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
--On 27 October 2011 14:54:48 +0100 Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:>> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > If this is an HVM guest,It is> you should use hda here because you want to > make sure that an emulated IDE disk is created for you as well.On Xen3.3 (and apparently on Xen 4) this happens anyway> Also if /tmp/centos-pvd.img is a raw file, you might as well use file: > rather than tap:aio.There seems to be some doubt (see Ian''s message) about whether this changes the backend driver that is used. The final deployment application is tap:aio with a block device, so that''s why we''re doing this. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
--On 27 October 2011 15:10:45 +0100 Ian Campbell <Ian.Campbell@citrix.com> wrote:>> We will in production use raw block devices, so that''s ok. We were >> just using files for testing. Is the same restriction there is >> Xen3.3, because it works there (possibly for the reason below). > > The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a > big factor. blkback.ko is in both so you should be ok. For testing you > could try using loopback like Stefano suggests.So, with apologies for being dense, what disk line should we be using to use blkback.ko? I''m assuming this is faster than the qemu based backend? (i.e. what you think xl may fall back to if blktap is not available) -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 14:19 UTC
Re: [Xen-devel] PV drivers on HVM using Xen 4.1.1
On Thu, 27 Oct 2011, Alex Bligh wrote:> > you should use hda here because you want to > > make sure that an emulated IDE disk is created for you as well. > > On Xen3.3 (and apparently on Xen 4) this happens anyway >Yes, but I wouldn''t rely on that: if you specify xvda it means you want *only* a pv disk. However if you don''t have any IDE disks configured, qemu realizes that you made a mistake in the config and setup one for you anyway.> > Also if /tmp/centos-pvd.img is a raw file, you might as well use file: > > rather than tap:aio. > > There seems to be some doubt (see Ian''s message) about whether this > changes the backend driver that is used. The final deployment application > is tap:aio with a block device, so that''s why we''re doing this.If you are using XL, no matter if you specify tap:aio or file:, you are going to get qemu as disk backend if you are missing blktap. There is nothing wrong with that, except that qemu in 4.1 doesn''t support linux aio so the performances are not very good. I am not sure which one is better: blkback on a loop device or qemu without linux aio, they are both rather slow. In order to make it fast you can: - use a dom0 kernel that provides blktap; - use LVM with blkback; - use upstream qemu with linux aio as device model and/or block backend. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:16 +0100, Alex Bligh wrote:> > --On 27 October 2011 15:10:45 +0100 Ian Campbell <Ian.Campbell@citrix.com> > wrote: > > >> We will in production use raw block devices, so that''s ok. We were > >> just using files for testing. Is the same restriction there is > >> Xen3.3, because it works there (possibly for the reason below). > > > > The 2.6.18 kernel has blktap.ko and 3.0 does not, which is going to be a > > big factor. blkback.ko is in both so you should be ok. For testing you > > could try using loopback like Stefano suggests. > > So, with apologies for being dense, what disk line should we be using > to use blkback.ko? I''m assuming this is faster than the qemu based > backend? (i.e. what you think xl may fall back to if blktap is not > available)I think just "/dev/something,,hda" (or xvda) The disk syntax is documented in docs/misc/xl-disk-configuration.txt, all this stuff with phy: and tap:aoi: prefixes is really legacy for compatibility with xend, with xend you might have used "phy:/dev/something...." IIRC. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:11 +0100, Alex Bligh wrote:> > --On 27 October 2011 14:54:48 +0100 Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > >> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > > > If this is an HVM guest, > > It is > > > you should use hda here because you want to > > make sure that an emulated IDE disk is created for you as well. > > On Xen3.3 (and apparently on Xen 4) this happens anyway > > > Also if /tmp/centos-pvd.img is a raw file, you might as well use file: > > rather than tap:aio. > > There seems to be some doubt (see Ian''s message) about whether this > changes the backend driver that is used. The final deployment application > is tap:aio with a block device, so that''s why we''re doing this.tap:aio on a raw block device is not a useful combination. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:19 +0100, Stefano Stabellini wrote:> On Thu, 27 Oct 2011, Alex Bligh wrote: > > > you should use hda here because you want to > > > make sure that an emulated IDE disk is created for you as well. > > > > On Xen3.3 (and apparently on Xen 4) this happens anyway > > > > Yes, but I wouldn''t rely on that: if you specify xvda it means you want > *only* a pv disk. However if you don''t have any IDE disks configured, > qemu realizes that you made a mistake in the config and setup one for > you anyway.Specifically _some_ qemu''s realize...> > > Also if /tmp/centos-pvd.img is a raw file, you might as well use file: > > > rather than tap:aio. > > > > There seems to be some doubt (see Ian''s message) about whether this > > changes the backend driver that is used. The final deployment application > > is tap:aio with a block device, so that''s why we''re doing this. > > If you are using XL, no matter if you specify tap:aio or file:, you are > going to get qemu as disk backend if you are missing blktap. > There is nothing wrong with that, except that qemu in 4.1 doesn''t > support linux aio so the performances are not very good. I am not sure > which one is better: blkback on a loop device or qemu without linux aio, > they are both rather slow. > > In order to make it fast you can: > > - use a dom0 kernel that provides blktap; > > - use LVM with blkback; > > - use upstream qemu with linux aio as device model and/or block backend.Alex''s real deployments use blkback on raw block devices so I think he is ok. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
--On 27 October 2011 15:19:16 +0100 Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:>> There seems to be some doubt (see Ian''s message) about whether this >> changes the backend driver that is used. The final deployment application >> is tap:aio with a block device, so that''s why we''re doing this. > > If you are using XL, no matter if you specify tap:aio or file:, you are > going to get qemu as disk backend if you are missing blktap. > There is nothing wrong with that, except that qemu in 4.1 doesn''t > support linux aio so the performances are not very good. I am not sure > which one is better: blkback on a loop device or qemu without linux aio, > they are both rather slow.I''m not sure I understand that. blkback on a loop device implies we can just use blkback on a real device. We are using a real device anyway in production (the file is just for testing). So, should we be using tap:aio:/dev/... for a block device for speed?> In order to make it fast you can: > > - use a dom0 kernel that provides blktap; > > - use LVM with blkback; > > - use upstream qemu with linux aio as device model and/or block backend.I /think/ you mean only if we are using a file, so that shouldn''t be relevant. Correct? -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 14:51 UTC
Re: [Xen-devel] PV drivers on HVM using Xen 4.1.1
On Thu, 27 Oct 2011, Alex Bligh wrote:> --On 27 October 2011 15:19:16 +0100 Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > >> There seems to be some doubt (see Ian''s message) about whether this > >> changes the backend driver that is used. The final deployment application > >> is tap:aio with a block device, so that''s why we''re doing this. > > > > If you are using XL, no matter if you specify tap:aio or file:, you are > > going to get qemu as disk backend if you are missing blktap. > > There is nothing wrong with that, except that qemu in 4.1 doesn''t > > support linux aio so the performances are not very good. I am not sure > > which one is better: blkback on a loop device or qemu without linux aio, > > they are both rather slow. > > I''m not sure I understand that. blkback on a loop device implies we > can just use blkback on a real device. We are using a real device > anyway in production (the file is just for testing).Ahh, sorry, I missed that you are using real devices in production.> So, should we be using tap:aio:/dev/... for a block device for > speed?You should use phy:/dev/whatever. At this point I suggest you setup loop devices for testing to be coherent.> > In order to make it fast you can: > > > > - use a dom0 kernel that provides blktap; > > > > - use LVM with blkback; > > > > - use upstream qemu with linux aio as device model and/or block backend. > > I /think/ you mean only if we are using a file, so that shouldn''t > be relevant. Correct?Correct. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:38 +0100, Alex Bligh wrote:> > --On 27 October 2011 15:19:16 +0100 Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > >> There seems to be some doubt (see Ian''s message) about whether this > >> changes the backend driver that is used. The final deployment application > >> is tap:aio with a block device, so that''s why we''re doing this. > > > > If you are using XL, no matter if you specify tap:aio or file:, you are > > going to get qemu as disk backend if you are missing blktap. > > There is nothing wrong with that, except that qemu in 4.1 doesn''t > > support linux aio so the performances are not very good. I am not sure > > which one is better: blkback on a loop device or qemu without linux aio, > > they are both rather slow. > > I''m not sure I understand that. blkback on a loop device implies we > can just use blkback on a real device. We are using a real device > anyway in production (the file is just for testing). > > So, should we be using tap:aio:/dev/... for a block device for > speed?tap:aio does not have a speed advantage -- in fact quite the opposite. tap: gives you the flexibility to use non-raw block devices and structured disk image types but is nothing like as fast as using a raw block device with blkback. The aio: part is an internal implementation detail of the tap: stuff which should never have been exposed to the user. Ian.> > > In order to make it fast you can: > > > > - use a dom0 kernel that provides blktap; > > > > - use LVM with blkback; > > > > - use upstream qemu with linux aio as device model and/or block backend. > > I /think/ you mean only if we are using a file, so that shouldn''t > be relevant. Correct? >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
--On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote:> I think I''ve found one reason why we can''t get PV block drivers on HVM > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux > domU kernel. > > We are using a line like: > > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > We are using effectively a standard 3.0 kernel. Config options including > the word XEN are below. > > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > start in Xen3.The magic incantation to fix this here was: modprobe xen_gntdev which for some reason is not modprobed in the startup script (Ubuntu one anyway). We now see pv block devices on hvm. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano, --On 27 October 2011 15:51:54 +0100 Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:>> So, should we be using tap:aio:/dev/... for a block device for >> speed? > > You should use phy:/dev/whatever. > At this point I suggest you setup loop devices for testing to be > coherent.Thanks. As per my last message, it looks like we have discovered the fundamental cause of the PV drivers not appearing (at least for xenlinux domUs). -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 27.10.2011 16:57, Alex Bligh wrote:> > > --On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote: > >> I think I''ve found one reason why we can''t get PV block drivers on HVM >> domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux >> domU kernel. >> >> We are using a line like: >> >> disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] >> >> We are using effectively a standard 3.0 kernel. Config options including >> the word XEN are below. >> >> We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does >> start in Xen3. > > The magic incantation to fix this here was: > modprobe xen_gntdev > which for some reason is not modprobed in the startup script (Ubuntu > one anyway). > > We now see pv block devices on hvm. >Hm, you mean in dom0? Strangely I do not have this loaded and still get pv disks on hvm... -Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-10-27 at 15:57 +0100, Alex Bligh wrote:> > --On 27 October 2011 13:56:41 +0100 Alex Bligh <alex@alex.org.uk> wrote: > > > I think I''ve found one reason why we can''t get PV block drivers on HVM > > domUs working on Xen 4.1.1 - whether we use a pvops or a xenlinux > > domU kernel. > > > > We are using a line like: > > > > disk = [ "tap:aio:/tmp/centos-pvd.img,xvda,w" ] > > > > We are using effectively a standard 3.0 kernel. Config options including > > the word XEN are below. > > > > We do not see blktapctrl starting in Xen 4.1.1 dom0, but it does > > start in Xen3. > > The magic incantation to fix this here was: > modprobe xen_gntdev > which for some reason is not modprobed in the startup script (Ubuntu > one anyway).I think 3.1 has patches to fix this in general (although I''m not sure about this specific case).> We now see pv block devices on hvm.If you are using gntdev then are you running a qemu to provide the backend? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-27 15:16 UTC
Re: [Xen-devel] PV drivers on HVM using Xen 4.1.1
On Thu, Oct 27, 2011 at 04:01:13PM +0100, Alex Bligh wrote:> Stefano, > > --On 27 October 2011 15:51:54 +0100 Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > >>So, should we be using tap:aio:/dev/... for a block device for > >>speed? > > > >You should use phy:/dev/whatever. > >At this point I suggest you setup loop devices for testing to be > >coherent.Or you can use Logical Volumes and dump your .img files in them and do: dd if=/mnt/ViP_customer.img of=/dev/vg_guest/VIP_customer bs=1MB and in your configuration file: disk = [''phy:/dev/vg_guest/VIP_customer,xvda,w''] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
--On 27 October 2011 17:09:16 +0200 Stefan Bader <stefan.bader@canonical.com> wrote:>> We now see pv block devices on hvm. >> > Hm, you mean in dom0? Strangely I do not have this loaded and still get > pv disks on hvm...I mean that without doing modprobe xen_gntdev on dom0, I do not see pv block devices on hvm in a domU. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian, --On 27 October 2011 16:13:20 +0100 Ian Campbell <Ian.Campbell@citrix.com> wrote:> If you are using gntdev then are you running a qemu to provide the > backend?We are using tap:aio:/path/to/file which I think we have established would fall back to qemu in the absence of blktap, as blkback only works with block devices. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel