Hello,
I am encountering an interesting problem when using PCI-hotplug to assign
devices to a VM (HVM) in a Xen environment (SLES 11 SP2).
1. If the VM is powered on before the PCI device is brought on-line, the guest
OS is unable to see the device (I have tried using
the acpiphp/pciehp drivers in the guest OS, as well as manually rescanning
the PCI bus from within the VM).
However, Xen reports that the hot-plug operation was successful. When I try
to verify the PCI device state from Xen''s perspective
by looking at a) the list of (remaining) assignable devices and b) list of
devices attached to the VM, the xm and xl tool-sets report
that the PCI device is indeed assigned to the VM.
2. If the VM is powered on after the PCI device (an FPGA in this case) is
brought on-line, the guest OS inside the VM can see and
configure the device as expected (using the appropriate driver for the
device).
Upon examining the output of /var/log/xen/qemu-dm-VMName.log, I see that in the
former case QEMU returned a failure, while in the latter case it reported
success (see below).
Case 1: VM was powered on ''before'' PCI device (FPGA) was
brought on-line
-------------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Error: couldn''t locate device in libpci
structures
Case 2: VM was powered on ''after'' PCI device (FPGA) was
brought on-line
----------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can''t open file /dev/xen/pci_iomul:
No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6000000)
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6100000)
pt_msi_setup: msi mapped with pirq 84 pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
I have seen this behavior when running both WindRiver and Ubuntu based operating
environments inside the VM. It doesn''t seem related to the type of
guest OS.
I have two questions, that I hoped someone on this list can answer for me.
A) Does Xen support case 1 (hotplugging a PCI device to a running VM that was
powered-on before the device was added to
the host)?
B) Why is Xen reporting that the hotplug operation was successful and that the
device is assigned to the VM, when QEMU reported
an error? Shouldn''t the Xen hotplug operation have failed as well?
Any help on this matter will be most appreciated.
Thanks.
-Anil
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
>>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote: > A) Does Xen support case 1 (hotplugging a PCI device to a running VM that > was powered-on before the device was added to > the host)?As far as I''m aware, only hotplug from the perspective of the guest is really supported/tested. But the issue here really seems less Xen related, but more towards libpci behavior (as according to qemu this is where the device isn''t known). As that''s a component Xen/qemu-dm simply builds upon, failure there (e.g. lack of hotplug awareness) would surface as a Xen misbehavior. Quickly going through the list of provided symbols of libpci.so doesn''t show anything that looks hotplug related (e.g. some sort of notification registration). But of course it''s possible that I''m overlooking something here, and that in fact qemu-dm''s interaction with libpci is simply incomplete. Jan
Thanks for the reply. After encountering this problem, I was afraid that only VM-level hotplug is supported by Xen (and not true hotplug at the host-level). Can someone kindly confirm that this is the case? Our current effort depends on host-level hotplug of a PCI device that can (later) be assigned (hotplugged) to a running VM, so a definitive answer will help us to avoid going down the wrong path. At the same time, I still think its odd that the Xen xm pci-attach command returned success when the attempted hotplug operation actually failed (as reported by the error message printed in the qemu-dm log file). Also, as I have indicated in my earlier post, Xen actually believes that the PCI device is assigned to the target VM (verified this by using xm pci-list to find out the list of devices attached to a VM), which again is very misleading. -Anil -----Original Message----- From: Jan Beulich [mailto:JBeulich@suse.com] Sent: Tuesday, June 05, 2012 1:43 AM To: Anil Rao Cc: xen-devel Subject: Re: [Xen-devel] PCI hotplug related issues>>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote: > A) Does Xen support case 1 (hotplugging a PCI device to a running VM > that was powered-on before the device was added to > the host)?As far as I''m aware, only hotplug from the perspective of the guest is really supported/tested. But the issue here really seems less Xen related, but more towards libpci behavior (as according to qemu this is where the device isn''t known). As that''s a component Xen/qemu-dm simply builds upon, failure there (e.g. lack of hotplug awareness) would surface as a Xen misbehavior. Quickly going through the list of provided symbols of libpci.so doesn''t show anything that looks hotplug related (e.g. some sort of notification registration). But of course it''s possible that I''m overlooking something here, and that in fact qemu-dm''s interaction with libpci is simply incomplete. Jan
>>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote: > After encountering this problem, I was afraid that only VM-level hotplug is > supported by Xen (and not true hotplug at the host-level). Can someone kindly > confirm that this is the case? Our current effort depends on host-level > hotplug of a PCI device that can (later) be assigned (hotplugged) to a > running VM, so a definitive answer will help us to avoid going down the wrong > path.That is not a proper statement - host side holtplug is still supported, just that currently it may need to happen before the target VM gets created. Overcoming limitations like this is certainly desirable, but will obviously require someone to actually invest work - the question would hence be whether, given that you''re interested in this feature and posted on xen-devel, you wouldn''t also want to look into addressing this (provided my brief analysis of the situation is correct at all). Jan
On Tue, 2012-06-05 at 09:42 +0100, Jan Beulich wrote:> >>> On 04.06.12 at 21:00, Anil Rao <anil.rao@ericsson.com> wrote: > > A) Does Xen support case 1 (hotplugging a PCI device to a running VM that > > was powered-on before the device was added to > > the host)? > > As far as I''m aware, only hotplug from the perspective of the guest > is really supported/tested. > > But the issue here really seems less Xen related, but more towards > libpci behavior (as according to qemu this is where the device isn''t > known). As that''s a component Xen/qemu-dm simply builds upon, > failure there (e.g. lack of hotplug awareness) would surface as a > Xen misbehavior. Quickly going through the list of provided symbols > of libpci.so doesn''t show anything that looks hotplug related (e.g. > some sort of notification registration). But of course it''s possible > that I''m overlooking something here, and that in fact qemu-dm''s > interaction with libpci is simply incomplete.Your theory tallies with what I remember from looking at PCI hotplug using a stub qemu -- IIRC libpci scans the bus at start of day and has no capability to refresh the list of known devices after that. This means that on host hotplug (actually mini-os hotplug in the case I was looking at) the device doesn''t become known to qemu. Anyway the upshot is that AFAIK even for non-stub qemu either (or both) qemu and libpci need to be taught about host PCI hotplug somehow. Ian.
On Wed, 2012-06-06 at 08:36 +0100, Jan Beulich wrote:> >>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote: > > After encountering this problem, I was afraid that only VM-level hotplug is > > supported by Xen (and not true hotplug at the host-level). Can someone kindly > > confirm that this is the case? Our current effort depends on host-level > > hotplug of a PCI device that can (later) be assigned (hotplugged) to a > > running VM, so a definitive answer will help us to avoid going down the wrong > > path. > > That is not a proper statement - host side holtplug is still supported, > just that currently it may need to happen before the target VM gets > created. Overcoming limitations like this is certainly desirable, but will > obviously require someone to actually invest work - the question > would hence be whether, given that you''re interested in this feature > and posted on xen-devel, you wouldn''t also want to look into > addressing this (provided my brief analysis of the situation is correct > at all).Yes, patches to make this work would be gratefully received. As an aside, I wonder if upstream qemu does anything better here -- that would probably be the right place to look at adding this new feature in any case. Ian.
Thanks, Jan and Ian for your confirmation and suggestions. For the moment, I have incorporated a workaround that is allowing us to proceed with our particular usage of Xen based VMs. When I free up a little bit at work, I don''t mind taking a stab at getting this to work. I''ll keep the list posted. Anil -----Original Message----- From: Ian Campbell [mailto:Ian.Campbell@citrix.com] Sent: Wednesday, June 06, 2012 2:29 AM To: Jan Beulich Cc: Anil Rao; xen-devel Subject: Re: [Xen-devel] PCI hotplug related issues On Wed, 2012-06-06 at 08:36 +0100, Jan Beulich wrote:> >>> On 05.06.12 at 20:01, Anil Rao <anil.rao@ericsson.com> wrote: > > After encountering this problem, I was afraid that only VM-level > > hotplug is supported by Xen (and not true hotplug at the > > host-level). Can someone kindly confirm that this is the case? Our > > current effort depends on host-level hotplug of a PCI device that > > can (later) be assigned (hotplugged) to a running VM, so a > > definitive answer will help us to avoid going down the wrong path. > > That is not a proper statement - host side holtplug is still > supported, just that currently it may need to happen before the target > VM gets created. Overcoming limitations like this is certainly > desirable, but will obviously require someone to actually invest work > - the question would hence be whether, given that you''re interested in > this feature and posted on xen-devel, you wouldn''t also want to look > into addressing this (provided my brief analysis of the situation is > correct at all).Yes, patches to make this work would be gratefully received. As an aside, I wonder if upstream qemu does anything better here -- that would probably be the right place to look at adding this new feature in any case. Ian.