I am running Xen 4 and want users to be able to install arbitrary HVM images. We want both PV and emulated drivers to appear. The reason for this is for compatibility with images that were prepared for Xen 3.3, which assume emulated devices exist at boot time (particularly for running the boot loader). If I do not specify on the boot line xen_emul_unplug=never, then under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel unplugs the non-PV devices, which is undesirable in my application. I cannot arbitrarily change the kernel command line because it is in effect set by the user''s image. Therefore what I need to do is prevent Xen4 offering the facility to unplug the devices in the first place. Setting "(unpluggable 0)" in the config for the emulated NICs and disks does not appear to prevent them getting unplugged; what may be happening is that domU tries to unplug them (but fails) but still doesn''t use them. Is there a way to signal from dom0 that the kernel really shouldn''t unplug this stuff. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-26 13:49 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:> I am running Xen 4 and want users to be able to install arbitrary HVM > images. We want both PV and emulated drivers to appear. The reason > for this is for compatibility with images that were prepared for > Xen 3.3, which assume emulated devices exist at boot time (particularly > for running the boot loader). > > If I do not specify on the boot line xen_emul_unplug=never, then > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel > unplugs the non-PV devices, which is undesirable in my application. > > I cannot arbitrarily change the kernel command line because it is > in effect set by the user''s image. > > Therefore what I need to do is prevent Xen4 offering the facility > to unplug the devices in the first place. Setting "(unpluggable 0)" > in the config for the emulated NICs and disks does not appear > to prevent them getting unplugged; what may be happening is that > domU tries to unplug them (but fails) but still doesn''t use them. > Is there a way to signal from dom0 that the kernel really shouldn''t > unplug this stuff.xen_pci_platform=0 in your guest config should do it. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Oct-26 14:12 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: > > I am running Xen 4 and want users to be able to install arbitrary HVM > > images. We want both PV and emulated drivers to appear. The reason > > for this is for compatibility with images that were prepared for > > Xen 3.3, which assume emulated devices exist at boot time (particularly > > for running the boot loader). > > > > If I do not specify on the boot line xen_emul_unplug=never, then > > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel > > unplugs the non-PV devices, which is undesirable in my application. > > > > I cannot arbitrarily change the kernel command line because it is > > in effect set by the user''s image. > > > > Therefore what I need to do is prevent Xen4 offering the facility > > to unplug the devices in the first place. Setting "(unpluggable 0)" > > in the config for the emulated NICs and disks does not appear > > to prevent them getting unplugged; what may be happening is that > > domU tries to unplug them (but fails) but still doesn''t use them. > > Is there a way to signal from dom0 that the kernel really shouldn''t > > unplug this stuff. > > xen_pci_platform=0 in your guest config should do it. >It''s actually called: xen_platform_pci=0 :) -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alex Bligh
2011-Oct-26 14:47 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
Pasi, Conrad, --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote: >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: >> > I am running Xen 4 and want users to be able to install arbitrary HVM >> > images. We want both PV and emulated drivers to appear. The reason >> > for this is for compatibility with images that were prepared for >> > Xen 3.3, which assume emulated devices exist at boot time (particularly >> > for running the boot loader). >> > >> > If I do not specify on the boot line xen_emul_unplug=never, then >> > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel >> > unplugs the non-PV devices, which is undesirable in my application. >> > >> > I cannot arbitrarily change the kernel command line because it is >> > in effect set by the user''s image. >> > >> > Therefore what I need to do is prevent Xen4 offering the facility >> > to unplug the devices in the first place. Setting "(unpluggable 0)" >> > in the config for the emulated NICs and disks does not appear >> > to prevent them getting unplugged; what may be happening is that >> > domU tries to unplug them (but fails) but still doesn''t use them. >> > Is there a way to signal from dom0 that the kernel really shouldn''t >> > unplug this stuff. >> >> xen_pci_platform=0 in your guest config should do it. >> > > It''s actually called: > xen_platform_pci=0Thanks. Just to confirm, that will still allow HVM PV drivers to operate (as well)? -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-26 14:56 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote:> Pasi, Conrad, > > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote: > >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: > >> > I am running Xen 4 and want users to be able to install arbitrary HVM > >> > images. We want both PV and emulated drivers to appear. The reason > >> > for this is for compatibility with images that were prepared for > >> > Xen 3.3, which assume emulated devices exist at boot time (particularly > >> > for running the boot loader). > >> > > >> > If I do not specify on the boot line xen_emul_unplug=never, then > >> > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel > >> > unplugs the non-PV devices, which is undesirable in my application. > >> > > >> > I cannot arbitrarily change the kernel command line because it is > >> > in effect set by the user''s image. > >> > > >> > Therefore what I need to do is prevent Xen4 offering the facility > >> > to unplug the devices in the first place. Setting "(unpluggable 0)" > >> > in the config for the emulated NICs and disks does not appear > >> > to prevent them getting unplugged; what may be happening is that > >> > domU tries to unplug them (but fails) but still doesn''t use them. > >> > Is there a way to signal from dom0 that the kernel really shouldn''t > >> > unplug this stuff. > >> > >> xen_pci_platform=0 in your guest config should do it. > >> > > > > It''s actually called: > > xen_platform_pci=0 > > Thanks. Just to confirm, that will still allow HVM PV drivers to > operate (as well)?No, this will disable PV drivers. The decision to unplug is a kernel side decision and in PVHVM Linux kernels is not currently possible to have both types of devices by default due to the risk of dataloss if the guest is not correctly configured (i.e. the kernel can''t tell if it is mounting the same filesystem via two paths). The xen_emul_unplug option is the current way you can override this once you have confirmed that your guest configuration is not dangerous. I''m afraid this necessarily involves guest config and guest admin interaction. In principal we might be able to extend the unplug protocol (which would involve patches to qemu, the kernel(s) and the toolstack) to allow devices to be marked as being not necessary to unplug. Someone would have to send patches though and it would be opening up a way for people to lose data so we''d need to be careful. I''m sure that the unplug protocol is documented somewhere in the source tree but I can''t for the life of me find it :-( Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Oct-26 15:54 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, Oct 26, 2011 at 03:47:13PM +0100, Alex Bligh wrote:> Pasi, Conrad, > > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote: > >> On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote: >>> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: >>> > I am running Xen 4 and want users to be able to install arbitrary HVM >>> > images. We want both PV and emulated drivers to appear. The reason >>> > for this is for compatibility with images that were prepared for >>> > Xen 3.3, which assume emulated devices exist at boot time (particularly >>> > for running the boot loader). >>> > >>> > If I do not specify on the boot line xen_emul_unplug=never, then >>> > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel >>> > unplugs the non-PV devices, which is undesirable in my application. >>> > >>> > I cannot arbitrarily change the kernel command line because it is >>> > in effect set by the user''s image. >>> > >>> > Therefore what I need to do is prevent Xen4 offering the facility >>> > to unplug the devices in the first place. Setting "(unpluggable 0)" >>> > in the config for the emulated NICs and disks does not appear >>> > to prevent them getting unplugged; what may be happening is that >>> > domU tries to unplug them (but fails) but still doesn''t use them. >>> > Is there a way to signal from dom0 that the kernel really shouldn''t >>> > unplug this stuff. >>> >>> xen_pci_platform=0 in your guest config should do it. >>> >> >> It''s actually called: >> xen_platform_pci=0 > > Thanks. Just to confirm, that will still allow HVM PV drivers to > operate (as well)? >Nope. PVHVM drivers will require the xen pci platform device to operate. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Oct-26 15:59 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, Oct 26, 2011 at 03:56:44PM +0100, Ian Campbell wrote:> On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote: > > Pasi, Conrad, > > > > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote: > > >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: > > >> > I am running Xen 4 and want users to be able to install arbitrary HVM > > >> > images. We want both PV and emulated drivers to appear. The reason > > >> > for this is for compatibility with images that were prepared for > > >> > Xen 3.3, which assume emulated devices exist at boot time (particularly > > >> > for running the boot loader). > > >> > > > >> > If I do not specify on the boot line xen_emul_unplug=never, then > > >> > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel > > >> > unplugs the non-PV devices, which is undesirable in my application. > > >> > > > >> > I cannot arbitrarily change the kernel command line because it is > > >> > in effect set by the user''s image. > > >> > > > >> > Therefore what I need to do is prevent Xen4 offering the facility > > >> > to unplug the devices in the first place. Setting "(unpluggable 0)" > > >> > in the config for the emulated NICs and disks does not appear > > >> > to prevent them getting unplugged; what may be happening is that > > >> > domU tries to unplug them (but fails) but still doesn''t use them. > > >> > Is there a way to signal from dom0 that the kernel really shouldn''t > > >> > unplug this stuff. > > >> > > >> xen_pci_platform=0 in your guest config should do it. > > >> > > > > > > It''s actually called: > > > xen_platform_pci=0 > > > > Thanks. Just to confirm, that will still allow HVM PV drivers to > > operate (as well)? > > No, this will disable PV drivers. > > The decision to unplug is a kernel side decision and in PVHVM Linux > kernels is not currently possible to have both types of devices by > default due to the risk of dataloss if the guest is not correctly > configured (i.e. the kernel can''t tell if it is mounting the same > filesystem via two paths). The xen_emul_unplug option is the current way > you can override this once you have confirmed that your guest > configuration is not dangerous. I''m afraid this necessarily involves > guest config and guest admin interaction. > > In principal we might be able to extend the unplug protocol (which would > involve patches to qemu, the kernel(s) and the toolstack) to allow > devices to be marked as being not necessary to unplug. Someone would > have to send patches though and it would be opening up a way for people > to lose data so we''d need to be careful. > > I''m sure that the unplug protocol is documented somewhere in the source > tree but I can''t for the life of me find it :-( >"disable qemu PCI devices in HVM domains": http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=e7911109f4321e9ba0cc56a253b653600aa46bea "disable qemu PCI devices in HVM domains - protocol document": http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=bc5c93ede039525765a7aa98e1da3386efa1465f -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-26 16:22 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, 2011-10-26 at 16:59 +0100, Pasi Kärkkäinen wrote:> On Wed, Oct 26, 2011 at 03:56:44PM +0100, Ian Campbell wrote: > > On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote: > > > Pasi, Conrad, > > > > > > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > > > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote: > > > >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote: > > > >> > I am running Xen 4 and want users to be able to install arbitrary HVM > > > >> > images. We want both PV and emulated drivers to appear. The reason > > > >> > for this is for compatibility with images that were prepared for > > > >> > Xen 3.3, which assume emulated devices exist at boot time (particularly > > > >> > for running the boot loader). > > > >> > > > > >> > If I do not specify on the boot line xen_emul_unplug=never, then > > > >> > under some guest OS''s (e.g. Ubuntu Natty (2.6.38)), the domU kernel > > > >> > unplugs the non-PV devices, which is undesirable in my application. > > > >> > > > > >> > I cannot arbitrarily change the kernel command line because it is > > > >> > in effect set by the user''s image. > > > >> > > > > >> > Therefore what I need to do is prevent Xen4 offering the facility > > > >> > to unplug the devices in the first place. Setting "(unpluggable 0)" > > > >> > in the config for the emulated NICs and disks does not appear > > > >> > to prevent them getting unplugged; what may be happening is that > > > >> > domU tries to unplug them (but fails) but still doesn''t use them. > > > >> > Is there a way to signal from dom0 that the kernel really shouldn''t > > > >> > unplug this stuff. > > > >> > > > >> xen_pci_platform=0 in your guest config should do it. > > > >> > > > > > > > > It''s actually called: > > > > xen_platform_pci=0 > > > > > > Thanks. Just to confirm, that will still allow HVM PV drivers to > > > operate (as well)? > > > > No, this will disable PV drivers. > > > > The decision to unplug is a kernel side decision and in PVHVM Linux > > kernels is not currently possible to have both types of devices by > > default due to the risk of dataloss if the guest is not correctly > > configured (i.e. the kernel can''t tell if it is mounting the same > > filesystem via two paths). The xen_emul_unplug option is the current way > > you can override this once you have confirmed that your guest > > configuration is not dangerous. I''m afraid this necessarily involves > > guest config and guest admin interaction. > > > > In principal we might be able to extend the unplug protocol (which would > > involve patches to qemu, the kernel(s) and the toolstack) to allow > > devices to be marked as being not necessary to unplug. Someone would > > have to send patches though and it would be opening up a way for people > > to lose data so we''d need to be careful. > > > > I''m sure that the unplug protocol is documented somewhere in the source > > tree but I can''t for the life of me find it :-( > > > > "disable qemu PCI devices in HVM domains": > http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=e7911109f4321e9ba0cc56a253b653600aa46bea > > "disable qemu PCI devices in HVM domains - protocol document": > http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=bc5c93ede039525765a7aa98e1da3386efa1465fThanks, I found it and am just about to send a patch to add it to xen-unstable.hg... Ian.> > > -- Pasi > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alex Bligh
2011-Oct-26 16:25 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
Ian,> No, this will disable PV drivers.I can confirm that our testing illustrates this :-(> The decision to unplug is a kernel side decision and in PVHVM Linux > kernels is not currently possible to have both types of devices by > default due to the risk of dataloss if the guest is not correctly > configured (i.e. the kernel can''t tell if it is mounting the same > filesystem via two paths). The xen_emul_unplug option is the current way > you can override this once you have confirmed that your guest > configuration is not dangerous. I''m afraid this necessarily involves > guest config and guest admin interaction. > > In principal we might be able to extend the unplug protocol (which would > involve patches to qemu, the kernel(s) and the toolstack) to allow > devices to be marked as being not necessary to unplug. Someone would > have to send patches though and it would be opening up a way for people > to lose data so we''d need to be careful. > > I''m sure that the unplug protocol is documented somewhere in the source > tree but I can''t for the life of me find it :-(So, the issue is this. We have thousands (literally) of disks in use by third parties on xen 3.3. Some are Windows, some are ancient linux, some are modern linux, etc. The hypervisor has no way of whether the images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated) drivers. Indeed the most common linux case is that grub uses the emulated devices to load the kernel, then uses /dev/xvda as a root device, i.e. both are used (but not simultaneously). We need to have the xen pci stuff on, so PV drivers operate (in both new and old kernels). But as modern linux kernels detect the unplug functionality, they will unplug the emulated devices and then fail to boot because (for instance) under Xen3.3 using /dev/sdaX to access (say) your /boot partition worked perfectly well. What we need is a switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any Linux kernel will see the same set of devices. I cannot believe this is a unique requirement for people attempting to do a Xen3.3 to Xen4.1 migration. I think this is in xen_unplug_emulated_devices() in arch/x86/xen/platform-pci-unplug.c This uses check_platform_magic(), which I have appended. In order to avoid unplugging (without relying on the boot line), I need this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is irrelevant as xen_emul_unplug is 0 by assumption). I can achieve that by either (a) returning a bad magic number, (b) making the host ''blacklist'' the product (how does that work?) or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply returns a bad magic number as I don''t think XEN_IOPORT_MAGIC existed in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for PCI unplug. So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather make it return a different value) depending on a configuration option? If so, I am happy to submit a patch to do that. Or can I do this without a patch by "blacklisting" everything? (not sure how that is done). Out of interest, with a default guest Ubuntu Natty install CD, using the default Xen 4.1 settings, we are seeing the guest (a) unplugging the emulated devices (fine), then (b) failing to find the emulated devices, and (c) the install failing. Is that to be expected? -- Alex Bligh static int __init check_platform_magic(void) { short magic; char protocol; magic = inw(XEN_IOPORT_MAGIC); if (magic != XEN_IOPORT_MAGIC_VAL) { printk(KERN_ERR "Xen Platform PCI: unrecognised magic value\n"); return XEN_PLATFORM_ERR_MAGIC; } protocol = inb(XEN_IOPORT_PROTOVER); printk(KERN_DEBUG "Xen Platform PCI: I/O protocol version %d\n", protocol); switch (protocol) { case 1: outw(XEN_IOPORT_LINUX_PRODNUM, XEN_IOPORT_PRODNUM); outl(XEN_IOPORT_LINUX_DRVVER, XEN_IOPORT_DRVVER); if (inw(XEN_IOPORT_MAGIC) != XEN_IOPORT_MAGIC_VAL) { printk(KERN_ERR "Xen Platform: blacklisted by host\n"); return XEN_PLATFORM_ERR_BLACKLIST; } break; default: printk(KERN_WARNING "Xen Platform PCI: unknown I/O protocol version"); return XEN_PLATFORM_ERR_PROTOCOL; } return 0; } _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-26 16:43 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, 2011-10-26 at 17:25 +0100, Alex Bligh wrote:> Ian, > > > No, this will disable PV drivers. > > I can confirm that our testing illustrates this :-( > > > The decision to unplug is a kernel side decision and in PVHVM Linux > > kernels is not currently possible to have both types of devices by > > default due to the risk of dataloss if the guest is not correctly > > configured (i.e. the kernel can''t tell if it is mounting the same > > filesystem via two paths). The xen_emul_unplug option is the current way > > you can override this once you have confirmed that your guest > > configuration is not dangerous. I''m afraid this necessarily involves > > guest config and guest admin interaction. > > > > In principal we might be able to extend the unplug protocol (which would > > involve patches to qemu, the kernel(s) and the toolstack) to allow > > devices to be marked as being not necessary to unplug. Someone would > > have to send patches though and it would be opening up a way for people > > to lose data so we''d need to be careful. > > > > I''m sure that the unplug protocol is documented somewhere in the source > > tree but I can''t for the life of me find it :-( > > So, the issue is this. We have thousands (literally) of disks in use > by third parties on xen 3.3. Some are Windows, some are ancient linux, > some are modern linux, etc. The hypervisor has no way of whether the > images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated) > drivers. Indeed the most common linux case is that grub uses the > emulated devices to load the kernel, then uses /dev/xvda as a root > device, i.e. both are used (but not simultaneously). > > We need to have the xen pci stuff on, so PV drivers operate (in both > new and old kernels). But as modern linux kernels detect the unplug > functionality, they will unplug the emulated devices and then fail to > boot because (for instance) under Xen3.3 using /dev/sdaX to access > (say) your /boot partition worked perfectly well. What we need is a > switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any > Linux kernel will see the same set of devices. I cannot believe this > is a unique requirement for people attempting to do a Xen3.3 to > Xen4.1 migration.I''m a bit fuzzy on the details but I''m not sure what this has to do with the host, the device naming and behaviour on unplug are kernel side things, I''d expect that if /dev/sdaX as /boot worked on 3.3 it''ll work on 4.1 too. (I believe you that it doesn''t work, I''m just wondering aloud what I''m missing). Can you give us the specifics of a setup which fails, e.g. a complete guest cfg file, the kernel version, command line options, /etc/fstab, dmesg etc.> I think this is in xen_unplug_emulated_devices() in > arch/x86/xen/platform-pci-unplug.c > > This uses check_platform_magic(), which I have appended. In order > to avoid unplugging (without relying on the boot line), I need > this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is > irrelevant as xen_emul_unplug is 0 by assumption). > > I can achieve that by either (a) returning a bad magic number, > (b) making the host ''blacklist'' the product (how does that work?) > or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply > returns a bad magic number as I don''t think XEN_IOPORT_MAGIC existed > in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for > PCI unplug. > > So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather > make it return a different value) depending on a configuration option? > If so, I am happy to submit a patch to do that. Or can I do this > without a patch by "blacklisting" everything? (not sure how that is done).Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on the kernel side is what I was missing. It might make sense to have a guest level config option which disables these magic ports, i.e. makes them return 0xffff like they would have done in 3.3 (I think 0xffff is what you''ll get from an invalid port in general).> Out of interest, with a default guest Ubuntu Natty install CD, using the > default Xen 4.1 settings, we are seeing the guest (a) unplugging the > emulated devices (fine), then (b) failing to find the emulated devices, > and (c) the install failing. Is that to be expected?Sounds like an Ubuntu bug to me, but I don''t follow Ubuntu closely enough to know if it is known or not. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alex Bligh
2011-Oct-26 17:43 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
Ian,> I''m a bit fuzzy on the details but I''m not sure what this has to do with > the host, the device naming and behaviour on unplug are kernel side > things, I''d expect that if /dev/sdaX as /boot worked on 3.3 it''ll work > on 4.1 too. (I believe you that it doesn''t work, I''m just wondering > aloud what I''m missing). > > Can you give us the specifics of a setup which fails, e.g. a complete > guest cfg file, the kernel version, command line options, /etc/fstab, > dmesg etc.I am not avoiding answering your question (I will get you this) but what is /meant/ to happen in the following scenario: * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3 * No fancy boot options, xen_emul_unplug not set * No XEN_IOPORT_MAGIC implemented, so check_platform_magic() returns an error * Therefore xen_platform_pci_unplug=0 * Therefore blkfront etc. don''t init (probe returns -ENODEV) * Therefore OS boots with root=/dev/sda Now Xen 3.3 is upgraded to Xen 4 * Kernel boots, and XEN_IOPORT_MAGIC now exists * Therefore unplug occurs, and xen_platform_pci_unplug is non zero * Therefore blkfront etc. inits, and PV drivers start * OS still boots with root=/dev/sda Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX? If so, how in the code does this happen? If not, won''t the boot fail?> Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on > the kernel side is what I was missing. > > It might make sense to have a guest level config option which disables > these magic ports, i.e. makes them return 0xffff like they would have > done in 3.3 (I think 0xffff is what you''ll get from an invalid port in > general).Actually I don''t think this will work. If we do this, xen_plaftofm_pci_unplug will still be zero (as it''s only set on exit of the function after a successful unplug), and that''s enough to prevent blkfront and xenbus_probe_frontend from doing anything useful, so will effectively disable PV drivers even where they should be enabled.>> Out of interest, with a default guest Ubuntu Natty install CD, using the >> default Xen 4.1 settings, we are seeing the guest (a) unplugging the >> emulated devices (fine), then (b) failing to find the emulated devices, >> and (c) the install failing. Is that to be expected? > > Sounds like an Ubuntu bug to me, but I don''t follow Ubuntu closely > enough to know if it is known or not.We will investigate further. Currently we can''t seem to get /any/ distro using PV drivers, and only old ones using emulated drivers. -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan Bader
2011-Oct-27 08:02 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On 26.10.2011 18:43, Ian Campbell wrote:> On Wed, 2011-10-26 at 17:25 +0100, Alex Bligh wrote: >> Ian, >> >>> No, this will disable PV drivers. >> >> I can confirm that our testing illustrates this :-( >> >>> The decision to unplug is a kernel side decision and in PVHVM Linux >>> kernels is not currently possible to have both types of devices by >>> default due to the risk of dataloss if the guest is not correctly >>> configured (i.e. the kernel can''t tell if it is mounting the same >>> filesystem via two paths). The xen_emul_unplug option is the current way >>> you can override this once you have confirmed that your guest >>> configuration is not dangerous. I''m afraid this necessarily involves >>> guest config and guest admin interaction. >>> >>> In principal we might be able to extend the unplug protocol (which would >>> involve patches to qemu, the kernel(s) and the toolstack) to allow >>> devices to be marked as being not necessary to unplug. Someone would >>> have to send patches though and it would be opening up a way for people >>> to lose data so we''d need to be careful. >>> >>> I''m sure that the unplug protocol is documented somewhere in the source >>> tree but I can''t for the life of me find it :-( >> >> So, the issue is this. We have thousands (literally) of disks in use >> by third parties on xen 3.3. Some are Windows, some are ancient linux, >> some are modern linux, etc. The hypervisor has no way of whether the >> images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated) >> drivers. Indeed the most common linux case is that grub uses the >> emulated devices to load the kernel, then uses /dev/xvda as a root >> device, i.e. both are used (but not simultaneously). >> >> We need to have the xen pci stuff on, so PV drivers operate (in both >> new and old kernels). But as modern linux kernels detect the unplug >> functionality, they will unplug the emulated devices and then fail to >> boot because (for instance) under Xen3.3 using /dev/sdaX to access >> (say) your /boot partition worked perfectly well. What we need is a >> switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any >> Linux kernel will see the same set of devices. I cannot believe this >> is a unique requirement for people attempting to do a Xen3.3 to >> Xen4.1 migration. > > I''m a bit fuzzy on the details but I''m not sure what this has to do with > the host, the device naming and behaviour on unplug are kernel side > things, I''d expect that if /dev/sdaX as /boot worked on 3.3 it''ll work > on 4.1 too. (I believe you that it doesn''t work, I''m just wondering > aloud what I''m missing). > > Can you give us the specifics of a setup which fails, e.g. a complete > guest cfg file, the kernel version, command line options, /etc/fstab, > dmesg etc. > >> I think this is in xen_unplug_emulated_devices() in >> arch/x86/xen/platform-pci-unplug.c >> >> This uses check_platform_magic(), which I have appended. In order >> to avoid unplugging (without relying on the boot line), I need >> this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is >> irrelevant as xen_emul_unplug is 0 by assumption). >> >> I can achieve that by either (a) returning a bad magic number, >> (b) making the host ''blacklist'' the product (how does that work?) >> or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply >> returns a bad magic number as I don''t think XEN_IOPORT_MAGIC existed >> in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for >> PCI unplug. >> >> So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather >> make it return a different value) depending on a configuration option? >> If so, I am happy to submit a patch to do that. Or can I do this >> without a patch by "blacklisting" everything? (not sure how that is done). > > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on > the kernel side is what I was missing. > > It might make sense to have a guest level config option which disables > these magic ports, i.e. makes them return 0xffff like they would have > done in 3.3 (I think 0xffff is what you''ll get from an invalid port in > general). > >> Out of interest, with a default guest Ubuntu Natty install CD, using the >> default Xen 4.1 settings, we are seeing the guest (a) unplugging the >> emulated devices (fine), then (b) failing to find the emulated devices, >> and (c) the install failing. Is that to be expected? > > Sounds like an Ubuntu bug to me, but I don''t follow Ubuntu closely > enough to know if it is known or not. >At least one part is not Ubuntu specific. And that is that the unplug logic decides to unplug emulated devices based on having the pci and the blkfront driver *available* (built-in or module). But later on the blkfront driver ignores all devices that are not *named* in a way to map to the xvd major. Which leaves you without any usable devices when you named your disk hda in the config file and you do not prevent unplugging. The other part of the problem is that even when you name the disk xvda in the config file, the installer does not know about blkfront. This is Ubuntu specific and we either need to have blkfront built-in or put them into a special udeb which is a special handling just for the installer. Still I would love to see this unplug handling become a bit more obvious. If unplug was successful, then blkfront should not ignore the devices. Or maybe just make the config more what-you-write-is-what-you-get and having hd or sd there only gives you emulated devices and xvd gives you pv devices. -Stefan> Ian. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-27 09:27 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Wed, 2011-10-26 at 18:43 +0100, Alex Bligh wrote:> Ian, > > > I''m a bit fuzzy on the details but I''m not sure what this has to do with > > the host, the device naming and behaviour on unplug are kernel side > > things, I''d expect that if /dev/sdaX as /boot worked on 3.3 it''ll work > > on 4.1 too. (I believe you that it doesn''t work, I''m just wondering > > aloud what I''m missing). > > > > Can you give us the specifics of a setup which fails, e.g. a complete > > guest cfg file, the kernel version, command line options, /etc/fstab, > > dmesg etc. > > I am not avoiding answering your question (I will get you this) but > what is /meant/ to happen in the following scenario:Lets call this scenario A:> * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3 > * No fancy boot options, xen_emul_unplug not set > * No XEN_IOPORT_MAGIC implemented, so check_platform_magic() > returns an errorCorrect, specifically it returns XEN_PLATFORM_ERR_MAGIC.> * Therefore xen_platform_pci_unplug=0Correct. There is some special casing around XEN_PLATFORM_ERR_MAGIC but lets assume xen_emul_unplug=unnecessary has not been passed to the kernel so it does not take effect.> * Therefore blkfront etc. don''t init (probe returns > -ENODEV) > * Therefore OS boots with root=/dev/sdasda must be an emulated IDE device since the 2.6.37 kernel does not support PV devices with names other than xvd*. I think this is as expected.> Now Xen 3.3 is upgraded to Xen 4Lets call this scenario B:> * Kernel boots, and XEN_IOPORT_MAGIC now exists > * Therefore unplug occurs, and xen_platform_pci_unplug is non zero > * Therefore blkfront etc. inits, and PV drivers start > * OS still boots with root=/dev/sda > > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?No, there is no support for this in upstream kernels (in general all the old behaviours of Xen kernels where they would hijack other drivers device names are not upstreamble)> If so, how in the code does this happen? If not, won''t the boot fail?You need to be using root=/dev/xvda here for it to work. Or even better use root=UUID=thing or root=LABEL=thong (many distros do one of these by default these days). (I''d forgotten about the UUID=/LABEL= option til just now -- that might be the bit of magic which was missing to make this work)> > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on > > the kernel side is what I was missing. > > > > It might make sense to have a guest level config option which disables > > these magic ports, i.e. makes them return 0xffff like they would have > > done in 3.3 (I think 0xffff is what you''ll get from an invalid port in > > general). > > Actually I don''t think this will work. If we do this, > xen_plaftofm_pci_unplug will still be zero (as it''s only set on exit > of the function after a successful unplug), and that''s enough to > prevent blkfront and xenbus_probe_frontend from doing anything useful, > so will effectively disable PV drivers even where they should be enabled.Correct, this will take you back to scenario A, however if that is how the guest is configured (to use emulated devices) then this is what you wanted (or at least it is what the guest configuration is expecting). If the guest were configured to use PV devices then it would be using root=/dev/xvda. Such a configuration would have needed xen_emul_unplug=unnecessary in order to have worked on 3.3 before the upgrade and this option would be harmless but unnecessary on 4.1. If you were using UUID=/LABEL= then I think things would have worked in both cases (emulated on 3.3 and pv on 4.1) without additional kernel parameters. One detail worth mentioning is that if the guest is using PV drivers and expecting xvd* named devices then prior to the unplug the devices xvd[a-d] are also exposed as the IDE devices hd[a-d]. This is how the bootloader is still able to access the emulated device. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-27 13:13 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
> > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX? > > No, there is no support for this in upstream kernels (in general all the > old behaviours of Xen kernels where they would hijack other drivers > device names are not upstreamble)Wait a minute. I think these two: konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c Stefan Bader (2): xen-blkfront: Drop name and minor adjustments for emulated scsi devices xen-blkfront: Fix one off warning about name clash Made it happen.. as without them the Amazon EC2 would not boot? Details: " xen-blkfront: Drop name and minor adjustments for emulated scsi devices These were intended to avoid the namespace clash when representing emulated IDE and SCSI devices. However that seems to confuse users more than expected (a disk defined as sda becomes xvde). So for now go back to the scheme which does no adjustments. This will break when mixing IDE and SCSI names in the configuration of guests but should be by now expected" _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alex Bligh
2011-Oct-27 13:18 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
Ian, --On 27 October 2011 10:27:08 +0100 Ian Campbell <Ian.Campbell@citrix.com> wrote:>> I am not avoiding answering your question (I will get you this) but >> what is /meant/ to happen in the following scenario:(now answered in another email) :-)> Lets call this scenario A: > >> * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3 >> * No fancy boot options, xen_emul_unplug not set >> * No XEN_IOPORT_MAGIC implemented, so check_platform_magic() >> returns an error > > Correct, specifically it returns XEN_PLATFORM_ERR_MAGIC. > >> * Therefore xen_platform_pci_unplug=0 > > Correct. There is some special casing around XEN_PLATFORM_ERR_MAGIC but > lets assume xen_emul_unplug=unnecessary has not been passed to the > kernel so it does not take effect. > >> * Therefore blkfront etc. don''t init (probe returns >> -ENODEV) >> * Therefore OS boots with root=/dev/sda > > sda must be an emulated IDE device since the 2.6.37 kernel does not > support PV devices with names other than xvd*. > > I think this is as expected.Yes, correct. So an in image created for xen 3.3 will have root=/dev/sda>> Now Xen 3.3 is upgraded to Xen 4 > > Lets call this scenario B: > >> * Kernel boots, and XEN_IOPORT_MAGIC now exists >> * Therefore unplug occurs, and xen_platform_pci_unplug is non zero >> * Therefore blkfront etc. inits, and PV drivers start >> * OS still boots with root=/dev/sda >> >> Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX? > > No, there is no support for this in upstream kernels (in general all the > old behaviours of Xen kernels where they would hijack other drivers > device names are not upstreamble)I thought so (from reading the code)>> If so, how in the code does this happen? If not, won''t the boot fail? > > You need to be using root=/dev/xvda here for it to work. Or even better > use root=UUID=thing or root=LABEL=thong (many distros do one of these by > default these days). > > (I''d forgotten about the UUID=/LABEL= option til just now -- that might > be the bit of magic which was missing to make this work)Right, so the same image that worked fine on Xen 3 will not work if Xen 3 is upgraded to Xen 4 - it will require changes to the image. It''s not only root= on the command line, but also /etc/fstab (we have partitioned disks in our images). Sure, using UUIDs works, but unfortunately we have thousands of VMs run by users who, not possessing crystal balls, did not know the names of their block devices were to change :-) For completeness, there is a also a problem with certain versions of grub (pre grub2) where UUID config doesn''t work; whilst grub of course loads before the kernel and before the unplug, the grub config file still references a device which has been unplugged, so the next time grub is reconfigured, it can''t find the disk to write to.>> > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on >> > the kernel side is what I was missing. >> > >> > It might make sense to have a guest level config option which disables >> > these magic ports, i.e. makes them return 0xffff like they would have >> > done in 3.3 (I think 0xffff is what you''ll get from an invalid port in >> > general). >> >> Actually I don''t think this will work. If we do this, >> xen_plaftofm_pci_unplug will still be zero (as it''s only set on exit >> of the function after a successful unplug), and that''s enough to >> prevent blkfront and xenbus_probe_frontend from doing anything useful, >> so will effectively disable PV drivers even where they should be enabled. > > Correct, this will take you back to scenario A, however if that is how > the guest is configured (to use emulated devices) then this is what you > wanted (or at least it is what the guest configuration is expecting). > > If the guest were configured to use PV devices then it would be using > root=/dev/xvda. Such a configuration would have needed > xen_emul_unplug=unnecessary in order to have worked on 3.3 before the > upgrade and this option would be harmless but unnecessary on 4.1.Yes. The trouble is some guests are running Windows, some old xenlinux kernels, and some new kernels that talk to XEN_IOPORT_MAGIC. I need one config that works for all of them, as we have no way of telling them apart. If I disable XEN_IOPORT_MAGIC then new installs of newer operating systems will not run with PV drivers. I agree that these won''t run (without tweaked kernel boot lines) on Xen 3.3 with PV drivers either, but I don''t want to be stuck this way for ever. I /think/ what you are saying is that the on new kernels that talk to XEN_IOPORT_MAGIC, they will use /dev/sdX (emulated disks). If we upgrade to Xen4, these will all break, because we can''t distinguish them from the behaviour of a new kernel doing a new install, where we want the unplug to occur. The fix is either to fix the root device, or to pass xen_emul_unplug=unnecessary in which case it should continue to use to the emulated devices. It would have been really nice to have been able to pass something through XEN_IOPORT_MAGIC to effectively cause the same effect as xen_emul_unplug=unnecessary on the command line, but I don''t see how that can be done with code already in the wild - unless I''m wrong?> If you were using UUID=/LABEL= then I think things would have worked in > both cases (emulated on 3.3 and pv on 4.1) without additional kernel > parameters. > > One detail worth mentioning is that if the guest is using PV drivers and > expecting xvd* named devices then prior to the unplug the devices > xvd[a-d] are also exposed as the IDE devices hd[a-d]. This is how the > bootloader is still able to access the emulated device.Sure -- Alex Bligh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 13:27 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 27 Oct 2011, Konrad Rzeszutek Wilk wrote:> > > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX? > > > > No, there is no support for this in upstream kernels (in general all the > > old behaviours of Xen kernels where they would hijack other drivers > > device names are not upstreamble) > > Wait a minute. I think these two: > > konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c > Stefan Bader (2): > xen-blkfront: Drop name and minor adjustments for emulated scsi devices > xen-blkfront: Fix one off warning about name clash > > > Made it happen.. as without them the Amazon EC2 would not boot? > > Details: " xen-blkfront: Drop name and minor adjustments for emulated scsi devices > > These were intended to avoid the namespace clash when representing > emulated IDE and SCSI devices. However that seems to confuse users > more than expected (a disk defined as sda becomes xvde). > So for now go back to the scheme which does no adjustments. This > will break when mixing IDE and SCSI names in the configuration of > guests but should be by now expected"This commit should only cause sda to become xvda rather than xvde. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 13:35 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 27 Oct 2011, Alex Bligh wrote:> It would have been really nice to have been able to pass something > through XEN_IOPORT_MAGIC to effectively cause the same effect as > xen_emul_unplug=unnecessary on the command line, but I don''t see how > that can be done with code already in the wild - unless I''m wrong?You are correct: with the current linux code you need to change the linux command line options in order to be able to accomplish what you want. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-27 13:35 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 2011-10-27 at 14:13 +0100, Konrad Rzeszutek Wilk wrote:> > > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX? > > > > No, there is no support for this in upstream kernels (in general all the > > old behaviours of Xen kernels where they would hijack other drivers > > device names are not upstreamble) > > Wait a minute. I think these two: > > konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c > Stefan Bader (2): > xen-blkfront: Drop name and minor adjustments for emulated scsi devices > xen-blkfront: Fix one off warning about name clash > > > Made it happen.. as without them the Amazon EC2 would not boot?No, that was to do with not renaming hda devices into xvdHDa (or something like that) and instead calling them hda. I was talking about blkfront literally taking over the IDE major which is not something that''s upstream.> > Details: " xen-blkfront: Drop name and minor adjustments for emulated scsi devices > > These were intended to avoid the namespace clash when representing > emulated IDE and SCSI devices. However that seems to confuse users > more than expected (a disk defined as sda becomes xvde). > So for now go back to the scheme which does no adjustments. This > will break when mixing IDE and SCSI names in the configuration of > guests but should be by now expected"_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 13:42 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
I take we are still talking about PV on HVM guests here. On Thu, 27 Oct 2011, Stefan Bader wrote:> At least one part is not Ubuntu specific. And that is that the unplug logic > decides to unplug emulated devices based on having the pci and the blkfront > driver *available* (built-in or module). But later on the blkfront driver > ignores all devices that are not *named* in a way to map to the xvd major. > Which leaves you without any usable devices when you named your disk hda in > the config file and you do not prevent unplugging.If you name your disk hda (as you should), blkfront is going to create /dev/xvda in your guest. It is not ignoring your disk, it just using "xvd" to name the device in the guest.> Still I would love to see this unplug handling become a bit more obvious. If > unplug was successful, then blkfront should not ignore the devices. Or maybe > just make the config more what-you-write-is-what-you-get and having hd or sd > there only gives you emulated devices and xvd gives you pv devices.Yes, if the unplug is unsuccessful blkfront should not ignore the device: it is going to create a /dev/xvd* for you. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan Bader
2011-Oct-27 13:45 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On 27.10.2011 15:42, Stefano Stabellini wrote:> I take we are still talking about PV on HVM guests here. > > On Thu, 27 Oct 2011, Stefan Bader wrote: >> At least one part is not Ubuntu specific. And that is that the unplug logic >> decides to unplug emulated devices based on having the pci and the blkfront >> driver *available* (built-in or module). But later on the blkfront driver >> ignores all devices that are not *named* in a way to map to the xvd major. >> Which leaves you without any usable devices when you named your disk hda in >> the config file and you do not prevent unplugging. > > If you name your disk hda (as you should), blkfront is going to create > /dev/xvda in your guest. > It is not ignoring your disk, it just using "xvd" to name the device in > the guest. > > >> Still I would love to see this unplug handling become a bit more obvious. If >> unplug was successful, then blkfront should not ignore the devices. Or maybe >> just make the config more what-you-write-is-what-you-get and having hd or sd >> there only gives you emulated devices and xvd gives you pv devices. > > Yes, if the unplug is unsuccessful blkfront should not ignore the > device: it is going to create a /dev/xvd* for you.The problem I saw in my test was that in blkfront_probe the following case was hit when the device name was hda in the cfg: if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) { int major; if (!VDEV_IS_EXTENDED(vdevice)) major = BLKIF_MAJOR(vdevice); else major = XENVBD_MAJOR; if (major != XENVBD_MAJOR) { printk(KERN_INFO "%s: HVM does not support vbd %d as xen block device\n", __FUNCTION__, vdevice); return -ENODEV; } } So major is not XENVBD_MAJOR and the device is ignored. -Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan Bader
2011-Oct-27 13:48 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On 27.10.2011 15:42, Stefano Stabellini wrote:> I take we are still talking about PV on HVM guests here. > > On Thu, 27 Oct 2011, Stefan Bader wrote: >> At least one part is not Ubuntu specific. And that is that the unplug logic >> decides to unplug emulated devices based on having the pci and the blkfront >> driver *available* (built-in or module). But later on the blkfront driver >> ignores all devices that are not *named* in a way to map to the xvd major. >> Which leaves you without any usable devices when you named your disk hda in >> the config file and you do not prevent unplugging. > > If you name your disk hda (as you should), blkfront is going to create > /dev/xvda in your guest. > It is not ignoring your disk, it just using "xvd" to name the device in > the guest. > > >> Still I would love to see this unplug handling become a bit more obvious. If >> unplug was successful, then blkfront should not ignore the devices. Or maybe >> just make the config more what-you-write-is-what-you-get and having hd or sd >> there only gives you emulated devices and xvd gives you pv devices. > > Yes, if the unplug is unsuccessful blkfront should not ignore the > device: it is going to create a /dev/xvd* for you.Hm, did you mean unsuccessful? Because then I think it *should* ignore the pv disk. If it was successful then it *should not*. Or am I understanding this wrong?> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 14:08 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 27 Oct 2011, Stefan Bader wrote:> On 27.10.2011 15:42, Stefano Stabellini wrote: > > I take we are still talking about PV on HVM guests here. > > > > On Thu, 27 Oct 2011, Stefan Bader wrote: > >> At least one part is not Ubuntu specific. And that is that the unplug logic > >> decides to unplug emulated devices based on having the pci and the blkfront > >> driver *available* (built-in or module). But later on the blkfront driver > >> ignores all devices that are not *named* in a way to map to the xvd major. > >> Which leaves you without any usable devices when you named your disk hda in > >> the config file and you do not prevent unplugging. > > > > If you name your disk hda (as you should), blkfront is going to create > > /dev/xvda in your guest. > > It is not ignoring your disk, it just using "xvd" to name the device in > > the guest. > > > > > >> Still I would love to see this unplug handling become a bit more obvious. If > >> unplug was successful, then blkfront should not ignore the devices. Or maybe > >> just make the config more what-you-write-is-what-you-get and having hd or sd > >> there only gives you emulated devices and xvd gives you pv devices. > > > > Yes, if the unplug is unsuccessful blkfront should not ignore the > > device: it is going to create a /dev/xvd* for you. > > The problem I saw in my test was that in blkfront_probe the following case was > hit when the device name was hda in the cfg: > > if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) { > int major; > > if (!VDEV_IS_EXTENDED(vdevice)) > major = BLKIF_MAJOR(vdevice); > else > major = XENVBD_MAJOR; > > if (major != XENVBD_MAJOR) { > printk(KERN_INFO > "%s: HVM does not support vbd %d > as xen block device\n", > __FUNCTION__, vdevice); > return -ENODEV; > } > } > > So major is not XENVBD_MAJOR and the device is ignored. >Why are you passing xen_emul_unplug=unnecessary? The idea is that if you pass that option you have to use the emulated path to access your disk unless you esplicitely add an xvd disk to your config. In the normal case, if you don''t specify xen_emul_unplug, you are going to get xvda for your hda disk. Then if you use LABEL or UUID in your root= argument, you don''t need to change any configuration to make it work. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 14:09 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 27 Oct 2011, Stefan Bader wrote:> On 27.10.2011 15:42, Stefano Stabellini wrote: > > I take we are still talking about PV on HVM guests here. > > > > On Thu, 27 Oct 2011, Stefan Bader wrote: > >> At least one part is not Ubuntu specific. And that is that the unplug logic > >> decides to unplug emulated devices based on having the pci and the blkfront > >> driver *available* (built-in or module). But later on the blkfront driver > >> ignores all devices that are not *named* in a way to map to the xvd major. > >> Which leaves you without any usable devices when you named your disk hda in > >> the config file and you do not prevent unplugging. > > > > If you name your disk hda (as you should), blkfront is going to create > > /dev/xvda in your guest. > > It is not ignoring your disk, it just using "xvd" to name the device in > > the guest. > > > > > >> Still I would love to see this unplug handling become a bit more obvious. If > >> unplug was successful, then blkfront should not ignore the devices. Or maybe > >> just make the config more what-you-write-is-what-you-get and having hd or sd > >> there only gives you emulated devices and xvd gives you pv devices. > > > > Yes, if the unplug is unsuccessful blkfront should not ignore the > > device: it is going to create a /dev/xvd* for you. > > Hm, did you mean unsuccessful? Because then I think it *should* ignore the pv > disk. If it was successful then it *should not*. Or am I understanding this wrong?Yes, sorry for the misspell. If the unplug is successful then it should not ignore the disk. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan Bader
2011-Oct-27 14:13 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On 27.10.2011 16:08, Stefano Stabellini wrote:> On Thu, 27 Oct 2011, Stefan Bader wrote: >> On 27.10.2011 15:42, Stefano Stabellini wrote: >>> I take we are still talking about PV on HVM guests here. >>> >>> On Thu, 27 Oct 2011, Stefan Bader wrote: >>>> At least one part is not Ubuntu specific. And that is that the unplug logic >>>> decides to unplug emulated devices based on having the pci and the blkfront >>>> driver *available* (built-in or module). But later on the blkfront driver >>>> ignores all devices that are not *named* in a way to map to the xvd major. >>>> Which leaves you without any usable devices when you named your disk hda in >>>> the config file and you do not prevent unplugging. >>> >>> If you name your disk hda (as you should), blkfront is going to create >>> /dev/xvda in your guest. >>> It is not ignoring your disk, it just using "xvd" to name the device in >>> the guest. >>> >>> >>>> Still I would love to see this unplug handling become a bit more obvious. If >>>> unplug was successful, then blkfront should not ignore the devices. Or maybe >>>> just make the config more what-you-write-is-what-you-get and having hd or sd >>>> there only gives you emulated devices and xvd gives you pv devices. >>> >>> Yes, if the unplug is unsuccessful blkfront should not ignore the >>> device: it is going to create a /dev/xvd* for you. >> >> The problem I saw in my test was that in blkfront_probe the following case was >> hit when the device name was hda in the cfg: >> >> if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) { >> int major; >> >> if (!VDEV_IS_EXTENDED(vdevice)) >> major = BLKIF_MAJOR(vdevice); >> else >> major = XENVBD_MAJOR; >> >> if (major != XENVBD_MAJOR) { >> printk(KERN_INFO >> "%s: HVM does not support vbd %d >> as xen block device\n", >> __FUNCTION__, vdevice); >> return -ENODEV; >> } >> } >> >> So major is not XENVBD_MAJOR and the device is ignored. >> > > Why are you passing xen_emul_unplug=unnecessary? > The idea is that if you pass that option you have to use the emulated > path to access your disk unless you esplicitely add an xvd disk to your > config. > > In the normal case, if you don''t specify xen_emul_unplug, you are going > to get xvda for your hda disk. > Then if you use LABEL or UUID in your root= argument, you don''t need to > change any configuration to make it work.Because of the above I have to specify the xen_emul_unplug to get *any* device. If I do not specify anything, then (because pci and pv driver are available) the emulated disk gets unplugged. Ok. But then the probe does not give me any xvd because the major does not match. The only form I could make this work was: use xvda in the config file. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Oct-27 14:25 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On Thu, 27 Oct 2011, Stefan Bader wrote:> On 27.10.2011 16:08, Stefano Stabellini wrote: > > On Thu, 27 Oct 2011, Stefan Bader wrote: > >> On 27.10.2011 15:42, Stefano Stabellini wrote: > >>> I take we are still talking about PV on HVM guests here. > >>> > >>> On Thu, 27 Oct 2011, Stefan Bader wrote: > >>>> At least one part is not Ubuntu specific. And that is that the unplug logic > >>>> decides to unplug emulated devices based on having the pci and the blkfront > >>>> driver *available* (built-in or module). But later on the blkfront driver > >>>> ignores all devices that are not *named* in a way to map to the xvd major. > >>>> Which leaves you without any usable devices when you named your disk hda in > >>>> the config file and you do not prevent unplugging. > >>> > >>> If you name your disk hda (as you should), blkfront is going to create > >>> /dev/xvda in your guest. > >>> It is not ignoring your disk, it just using "xvd" to name the device in > >>> the guest. > >>> > >>> > >>>> Still I would love to see this unplug handling become a bit more obvious. If > >>>> unplug was successful, then blkfront should not ignore the devices. Or maybe > >>>> just make the config more what-you-write-is-what-you-get and having hd or sd > >>>> there only gives you emulated devices and xvd gives you pv devices. > >>> > >>> Yes, if the unplug is unsuccessful blkfront should not ignore the > >>> device: it is going to create a /dev/xvd* for you. > >> > >> The problem I saw in my test was that in blkfront_probe the following case was > >> hit when the device name was hda in the cfg: > >> > >> if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) { > >> int major; > >> > >> if (!VDEV_IS_EXTENDED(vdevice)) > >> major = BLKIF_MAJOR(vdevice); > >> else > >> major = XENVBD_MAJOR; > >> > >> if (major != XENVBD_MAJOR) { > >> printk(KERN_INFO > >> "%s: HVM does not support vbd %d > >> as xen block device\n", > >> __FUNCTION__, vdevice); > >> return -ENODEV; > >> } > >> } > >> > >> So major is not XENVBD_MAJOR and the device is ignored. > >> > > > > Why are you passing xen_emul_unplug=unnecessary? > > The idea is that if you pass that option you have to use the emulated > > path to access your disk unless you esplicitely add an xvd disk to your > > config. > > > > In the normal case, if you don''t specify xen_emul_unplug, you are going > > to get xvda for your hda disk. > > Then if you use LABEL or UUID in your root= argument, you don''t need to > > change any configuration to make it work. > > Because of the above I have to specify the xen_emul_unplug to get *any* device.Why? If you do *not* specify xen_emul_unplug, you should get /dev/xvda (provided you have blkfront available and hda in your disk config line).> If I do not specify anything, then (because pci and pv driver are available) the > emulated disk gets unplugged. Ok. But then the probe does not give me any xvd > because the major does not match.that is only because you passed xen_emul_unplug=unnecessary; you shouldn''t do that> The only form I could make this work was: use xvda in the config file.you should use hda in the disk config line, and then configure the guest kernel root= argument with /dev/xvda1 directly or LABEL or UUID (better). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan Bader
2011-Oct-27 14:46 UTC
Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38
On 27.10.2011 16:25, Stefano Stabellini wrote:> On Thu, 27 Oct 2011, Stefan Bader wrote: >> On 27.10.2011 16:08, Stefano Stabellini wrote: >>> On Thu, 27 Oct 2011, Stefan Bader wrote: >>>> On 27.10.2011 15:42, Stefano Stabellini wrote: >>>>> I take we are still talking about PV on HVM guests here. >>>>> >>>>> On Thu, 27 Oct 2011, Stefan Bader wrote: >>>>>> At least one part is not Ubuntu specific. And that is that the unplug logic >>>>>> decides to unplug emulated devices based on having the pci and the blkfront >>>>>> driver *available* (built-in or module). But later on the blkfront driver >>>>>> ignores all devices that are not *named* in a way to map to the xvd major. >>>>>> Which leaves you without any usable devices when you named your disk hda in >>>>>> the config file and you do not prevent unplugging. >>>>> >>>>> If you name your disk hda (as you should), blkfront is going to create >>>>> /dev/xvda in your guest. >>>>> It is not ignoring your disk, it just using "xvd" to name the device in >>>>> the guest. >>>>> >>>>> >>>>>> Still I would love to see this unplug handling become a bit more obvious. If >>>>>> unplug was successful, then blkfront should not ignore the devices. Or maybe >>>>>> just make the config more what-you-write-is-what-you-get and having hd or sd >>>>>> there only gives you emulated devices and xvd gives you pv devices. >>>>> >>>>> Yes, if the unplug is unsuccessful blkfront should not ignore the >>>>> device: it is going to create a /dev/xvd* for you. >>>> >>>> The problem I saw in my test was that in blkfront_probe the following case was >>>> hit when the device name was hda in the cfg: >>>> >>>> if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) { >>>> int major; >>>> >>>> if (!VDEV_IS_EXTENDED(vdevice)) >>>> major = BLKIF_MAJOR(vdevice); >>>> else >>>> major = XENVBD_MAJOR; >>>> >>>> if (major != XENVBD_MAJOR) { >>>> printk(KERN_INFO >>>> "%s: HVM does not support vbd %d >>>> as xen block device\n", >>>> __FUNCTION__, vdevice); >>>> return -ENODEV; >>>> } >>>> } >>>> >>>> So major is not XENVBD_MAJOR and the device is ignored. >>>> >>> >>> Why are you passing xen_emul_unplug=unnecessary? >>> The idea is that if you pass that option you have to use the emulated >>> path to access your disk unless you esplicitely add an xvd disk to your >>> config. >>> (and force the blkfront to get loaded but that is some other patches missing >>> In the normal case, if you don''t specify xen_emul_unplug, you are going >>> to get xvda for your hda disk. >>> Then if you use LABEL or UUID in your root= argument, you don''t need to >>> change any configuration to make it work. >> >> Because of the above I have to specify the xen_emul_unplug to get *any* device. > > Why? > If you do *not* specify xen_emul_unplug, you should get /dev/xvda > (provided you have blkfront available and hda in your disk config line). >Ah! Sorry, yes. I think I remember now how my confusion started. While trying to find the lost interrupt problem for the network devices I had two of them configured (8139cp and e1000) and during those tests there was no way I was able to get any xvd device. But that was because for some reason with both NICs the pci device would fail to get an IRQ, so xenbus was never started. Removing the second NIC made the pci device work, but then with allowing unplug, there was still some problem with the pv nic not working. So I probably ended up with the unnecessary, and when the disk finally worked when defined as xvda, and I was already too confused by other things to realize that that segment only applies for the unecessary case. :/> >> If I do not specify anything, then (because pci and pv driver are available) the >> emulated disk gets unplugged. Ok. But then the probe does not give me any xvd >> because the major does not match. > > that is only because you passed xen_emul_unplug=unnecessary; you > shouldn''t do that > >> The only form I could make this work was: use xvda in the config file. > > you should use hda in the disk config line, and then configure the guest > kernel root= argument with /dev/xvda1 directly or LABEL or UUID > (better)._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Possibly Parallel Threads
- Xen 4.x / Linux 3.x (dom0 and HVM domU) and NIC handling
- [PATCH] xen: initialize platform_pci even if xen_emul_unplug=never
- [PATCH] xen/pvhvm: If xen_platform_pci=0 is set don''t blow up.
- RFH: loopback & blktap(2) and CDROM
- Phoronix Xen vs KVM vs Virtualbox benchmark with Ubuntu 11.10