Hi, I am trying to assign a FPGA (which is a PCI device) to a running VM. The interesting thing here is that the FPGA is not present when the VM was powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line on the host by downloading its image to the device. So far my attempts at using Xen PCI-passthrough have not been successful. Although Xen reports that the pass-through operation was successful, which I have verified using the appropriate xl command, I am unable to see the FPGA from within the guest OS. I have tried both manual rescanning of the PCI bus inside the guest as well as making use of the pciehp driver. The only way I can (reliably) see the FPGA from within the guest, is to shutdown and restart the VM and then redo the hotplug operation. I want to avoid a VM reboot and was wondering if this is at all possible and if so what do I need to do to realize it. [The virtualization host is based on the default Xen setup that comes with SLES 11 SP2] Thanks for any help on this matter. -Anil _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Are you sure the guest OS has PCI hotplug support? On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@ericsson.com> wrote:> Hi, > > I am trying to assign a FPGA (which is a PCI device) to a running VM. The > interesting thing here is that the FPGA is not present when the VM was > powered-on. I.e. sometime after the VM was started, the FPGA was brought > on-line on the host by downloading its image to the device. > > So far my attempts at using Xen PCI-passthrough have not been successful. > Although Xen reports that the pass-through operation was successful, which > I have verified using the appropriate xl command, I am unable to see the > FPGA from within the guest OS. > > I have tried both manual rescanning of the PCI bus inside the guest as > well as making use of the pciehp driver. The only way I can (reliably) see > the FPGA from within the guest, is to shutdown and restart the VM and then > redo the hotplug operation. I want to avoid a VM reboot and was wondering > if this is at all possible and if so what do I need to do to realize it. > > [The virtualization host is based on the default Xen setup that comes with > SLES 11 SP2] > > Thanks for any help on this matter. > > -Anil > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI devices to the VM. The only one that doesn''t work is the FPGA device. Even for this one, the only time it doesn''t work, is if the VM was powered-on prior to the FPGA device being brought on-line on the host. If the FPGA was present before the VM was powered-on everything works as expected. It somehow seems like new devices that are introduced in the host, after a VM has been started on it, cannot be hotplugged into the VM. Thanks. -Anil ________________________________ From: theubaz@gmail.com [mailto:theubaz@gmail.com] On Behalf Of John Sherwood Sent: Wednesday, May 30, 2012 7:20 PM To: Anil Rao Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] Hotplugging FPGA to running VM Are you sure the guest OS has PCI hotplug support? On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote: Hi, I am trying to assign a FPGA (which is a PCI device) to a running VM. The interesting thing here is that the FPGA is not present when the VM was powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line on the host by downloading its image to the device. So far my attempts at using Xen PCI-passthrough have not been successful. Although Xen reports that the pass-through operation was successful, which I have verified using the appropriate xl command, I am unable to see the FPGA from within the guest OS. I have tried both manual rescanning of the PCI bus inside the guest as well as making use of the pciehp driver. The only way I can (reliably) see the FPGA from within the guest, is to shutdown and restart the VM and then redo the hotplug operation. I want to avoid a VM reboot and was wondering if this is at all possible and if so what do I need to do to realize it. [The virtualization host is based on the default Xen setup that comes with SLES 11 SP2] Thanks for any help on this matter. -Anil _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Just out of curiosity, what kind of *FPGA device* is this? Is it an FPGA development board or something already assembled and functioning that contains an FPGA device(s)? Well, just being curious... :)> Yes, the guest OS has PCI hotplug support. I am able to hotplug a > variety of PCI devices to the VM. The only one that doesn''t work is > the FPGA device. Even for this one, the only time it doesn''t work, is > if the VM was powered-on prior to the FPGA device being brought > on-line on the host. If the FPGA was present before the VM was > powered-on everything works as expected. > It somehow seems like new devices that are introduced in the host, > after a VM has been started on it, cannot be hotplugged into the VM. > Thanks. > -Anil > ------------------------------------------------------------------------ > *From:* theubaz@gmail.com [mailto:theubaz@gmail.com] *On Behalf Of > *John Sherwood > *Sent:* Wednesday, May 30, 2012 7:20 PM > *To:* Anil Rao > *Cc:* xen-users@lists.xensource.com > *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM > > Are you sure the guest OS has PCI hotplug support? > > On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@ericsson.com > <mailto:anil.rao@ericsson.com>> wrote: > > Hi, > I am trying to assign a FPGA (which is a PCI device) to a running > VM. The interesting thing here is that the FPGA is not present > when the VM was powered-on. I.e. sometime after the VM was > started, the FPGA was brought on-line on the host by downloading > its image to the device. > So far my attempts at using Xen PCI-passthrough have not been > successful. Although Xen reports that the pass-through operation > was successful, which I have verified using the appropriate xl > command, I am unable to see the FPGA from within the guest OS. > I have tried both manual rescanning of the PCI bus inside the > guest as well as making use of the pciehp driver. The only way I > can (reliably) see the FPGA from within the guest, is to shutdown > and restart the VM and then redo the hotplug operation. I want to > avoid a VM reboot and was wondering if this is at all possible and > if so what do I need to do to realize it. > [The virtualization host is based on the default Xen setup that > comes with SLES 11 SP2] > Thanks for any help on this matter. > -Anil > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
This is a proprietary FPGA device. Each control blade in our router chassis has
one of these FPGAs and its primary function is to control certain dedicated
hardware components on the blade that are responsible for establishing
connectivity with the high-speed inter-blade fabric on the router. I am sorry,
but I cannot divulge additional details regarding the FPGA.
Here is a more detailed description of my actual usage scenario.
1) The FPGA image must be downloaded onto the device before it can be enabled
for use.
2) The VM in question is responsible for performing step (1), which means that
the FPGA is not visible to Xen/Dom-0 when
the VM is powered on.
3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen
pci-hotplug to assign the device to the VM.
This operation completes successfully (as reported by the appropriate xl
command).
4) However, the guest OS inside the VM cannot see the FPGA device after step
(3).
5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the
FPGA device (and everything then proceeds normally).
The troublesome part here is that I need to shutdown and restart the VM. I would
like to avoid this and rely on Xen pci-hotplug to make the device visible to the
guest OS inside the VM.
Thanks.
-Anil
________________________________
From: xen-users-bounces@lists.xen.org [mailto:xen-users-bounces@lists.xen.org]
On Behalf Of cyberhawk001@gmail.com
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@lists.xen.org
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Just out of curiosity, what kind of FPGA device is this? Is it an FPGA
development board or something already assembled and functioning that contains
an FPGA device(s)?
Well, just being curious... :)
Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI
devices to the VM. The only one that doesn''t work is the FPGA device.
Even for this one, the only time it doesn''t work, is if the VM was
powered-on prior to the FPGA device being brought on-line on the host. If the
FPGA was present before the VM was powered-on everything works as expected.
It somehow seems like new devices that are introduced in the host, after a VM
has been started on it, cannot be hotplugged into the VM.
Thanks.
-Anil
________________________________
From: theubaz@gmail.com<mailto:theubaz@gmail.com>
[mailto:theubaz@gmail.com] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Are you sure the guest OS has PCI hotplug support?
On Wed, May 30, 2012 at 10:14 PM, Anil Rao
<anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote:
Hi,
I am trying to assign a FPGA (which is a PCI device) to a running VM. The
interesting thing here is that the FPGA is not present when the VM was
powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line
on the host by downloading its image to the device.
So far my attempts at using Xen PCI-passthrough have not been successful.
Although Xen reports that the pass-through operation was successful, which I
have verified using the appropriate xl command, I am unable to see the FPGA from
within the guest OS.
I have tried both manual rescanning of the PCI bus inside the guest as well as
making use of the pciehp driver. The only way I can (reliably) see the FPGA from
within the guest, is to shutdown and restart the VM and then redo the hotplug
operation. I want to avoid a VM reboot and was wondering if this is at all
possible and if so what do I need to do to realize it.
[The virtualization host is based on the default Xen setup that comes with SLES
11 SP2]
Thanks for any help on this matter.
-Anil
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org<mailto:Xen-users@lists.xen.org>
http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Since I am new to the xen-users list, I was wondering if my question is more
appropriate for the xen-devel list. Any thoughts?
Thanks.
-Anil
________________________________
From: xen-users-bounces@lists.xen.org [mailto:xen-users-bounces@lists.xen.org]
On Behalf Of Anil Rao
Sent: Thursday, May 31, 2012 11:25 AM
To: cyberhawk001@gmail.com; xen-users@lists.xen.org
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
This is a proprietary FPGA device. Each control blade in our router chassis has
one of these FPGAs and its primary function is to control certain dedicated
hardware components on the blade that are responsible for establishing
connectivity with the high-speed inter-blade fabric on the router. I am sorry,
but I cannot divulge additional details regarding the FPGA.
Here is a more detailed description of my actual usage scenario.
1) The FPGA image must be downloaded onto the device before it can be enabled
for use.
2) The VM in question is responsible for performing step (1), which means that
the FPGA is not visible to Xen/Dom-0 when
the VM is powered on.
3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen
pci-hotplug to assign the device to the VM.
This operation completes successfully (as reported by the appropriate xl
command).
4) However, the guest OS inside the VM cannot see the FPGA device after step
(3).
5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the
FPGA device (and everything then proceeds normally).
The troublesome part here is that I need to shutdown and restart the VM. I would
like to avoid this and rely on Xen pci-hotplug to make the device visible to the
guest OS inside the VM.
Thanks.
-Anil
________________________________
From: xen-users-bounces@lists.xen.org [mailto:xen-users-bounces@lists.xen.org]
On Behalf Of cyberhawk001@gmail.com
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@lists.xen.org
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Just out of curiosity, what kind of FPGA device is this? Is it an FPGA
development board or something already assembled and functioning that contains
an FPGA device(s)?
Well, just being curious... :)
Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI
devices to the VM. The only one that doesn''t work is the FPGA device.
Even for this one, the only time it doesn''t work, is if the VM was
powered-on prior to the FPGA device being brought on-line on the host. If the
FPGA was present before the VM was powered-on everything works as expected.
It somehow seems like new devices that are introduced in the host, after a VM
has been started on it, cannot be hotplugged into the VM.
Thanks.
-Anil
________________________________
From: theubaz@gmail.com<mailto:theubaz@gmail.com>
[mailto:theubaz@gmail.com] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Are you sure the guest OS has PCI hotplug support?
On Wed, May 30, 2012 at 10:14 PM, Anil Rao
<anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote:
Hi,
I am trying to assign a FPGA (which is a PCI device) to a running VM. The
interesting thing here is that the FPGA is not present when the VM was
powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line
on the host by downloading its image to the device.
So far my attempts at using Xen PCI-passthrough have not been successful.
Although Xen reports that the pass-through operation was successful, which I
have verified using the appropriate xl command, I am unable to see the FPGA from
within the guest OS.
I have tried both manual rescanning of the PCI bus inside the guest as well as
making use of the pciehp driver. The only way I can (reliably) see the FPGA from
within the guest, is to shutdown and restart the VM and then redo the hotplug
operation. I want to avoid a VM reboot and was wondering if this is at all
possible and if so what do I need to do to realize it.
[The virtualization host is based on the default Xen setup that comes with SLES
11 SP2]
Thanks for any help on this matter.
-Anil
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
If you''ve never seen this issue before and can''t reproduce it with any other PCI devices, I''d suggest trying the devel mailing list, but I''d first collect whatever logs you can that are salient to the issue beforehand. I''ve never had issues with PCI hot plug that were limited to a single device though, so I''m certainly drawing a blank here on what it could be. On Thu, May 31, 2012 at 8:06 PM, Anil Rao <anil.rao@ericsson.com> wrote:> ** > Since I am new to the xen-users list, I was wondering if my question is > more appropriate for the xen-devel list. Any thoughts? > > Thanks. > -Anil > > ------------------------------ > *From:* xen-users-bounces@lists.xen.org [mailto: > xen-users-bounces@lists.xen.org] *On Behalf Of *Anil Rao > *Sent:* Thursday, May 31, 2012 11:25 AM > *To:* cyberhawk001@gmail.com; xen-users@lists.xen.org > > *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM > > This is a proprietary FPGA device. Each control blade in our router > chassis has one of these FPGAs and its primary function is to control > certain dedicated hardware components on the blade that are responsible for > establishing connectivity with the high-speed inter-blade fabric on the > router. I am sorry, but I cannot divulge additional details regarding the > FPGA. > > Here is a more detailed description of my actual usage scenario. > > 1) The FPGA image must be downloaded onto the device before it can be > enabled for use. > 2) The VM in question is responsible for performing step (1), which means > that the FPGA is not visible to Xen/Dom-0 when > the VM is powered on. > 3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen > pci-hotplug to assign the device to the VM. > This operation completes successfully (as reported by the appropriate > xl command). > 4) However, the guest OS inside the VM cannot see the FPGA device after > step (3). > 5) If I shutdown the VM and re-start it, the guest OS inside the VM sees > the FPGA device (and everything then proceeds normally). > > The troublesome part here is that I need to shutdown and restart the VM. I > would like to avoid this and rely on Xen pci-hotplug to make the device > visible to the guest OS inside the VM. > > Thanks. > -Anil > > ------------------------------ > *From:* xen-users-bounces@lists.xen.org [mailto: > xen-users-bounces@lists.xen.org] *On Behalf Of *cyberhawk001@gmail.com > *Sent:* Thursday, May 31, 2012 10:49 AM > *To:* xen-users@lists.xen.org > *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM > > Just out of curiosity, what kind of *FPGA device* is this? Is it an FPGA > development board or something already assembled and functioning that > contains an FPGA device(s)? > > Well, just being curious... :) > > > > Yes, the guest OS has PCI hotplug support. I am able to hotplug a > variety of PCI devices to the VM. The only one that doesn''t work is the > FPGA device. Even for this one, the only time it doesn''t work, is if the VM > was powered-on prior to the FPGA device being brought on-line on the host. > If the FPGA was present before the VM was powered-on everything works as > expected. > > It somehow seems like new devices that are introduced in the host, after a > VM has been started on it, cannot be hotplugged into the VM. > > Thanks. > -Anil > > > ------------------------------ > *From:* theubaz@gmail.com [mailto:theubaz@gmail.com <theubaz@gmail.com>] > *On Behalf Of *John Sherwood > *Sent:* Wednesday, May 30, 2012 7:20 PM > *To:* Anil Rao > *Cc:* xen-users@lists.xensource.com > *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM > > Are you sure the guest OS has PCI hotplug support? > > On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@ericsson.com> wrote: > >> Hi, >> >> I am trying to assign a FPGA (which is a PCI device) to a running VM. The >> interesting thing here is that the FPGA is not present when the VM was >> powered-on. I.e. sometime after the VM was started, the FPGA was brought >> on-line on the host by downloading its image to the device. >> >> So far my attempts at using Xen PCI-passthrough have not been successful. >> Although Xen reports that the pass-through operation was successful, which >> I have verified using the appropriate xl command, I am unable to see the >> FPGA from within the guest OS. >> >> I have tried both manual rescanning of the PCI bus inside the guest as >> well as making use of the pciehp driver. The only way I can (reliably) see >> the FPGA from within the guest, is to shutdown and restart the VM and then >> redo the hotplug operation. I want to avoid a VM reboot and was wondering >> if this is at all possible and if so what do I need to do to realize it. >> >> [The virtualization host is based on the default Xen setup that comes >> with SLES 11 SP2] >> >> Thanks for any help on this matter. >> >> -Anil >> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Thanks, John, for your thoughts.
The problem is not about a particular PCI device. Rather, it seems to affect
several different types of devices.
What I am seeing is this. If a device is added to the host "after" a
VM has been started on it, any attempt to hotplug that device to the (already
running VM) results in the following.
From Xen''s point of view the hotplug operation is successful. The
guest OS in the VM, however, cannot see the device.
If the VM was started after the device was added to the host, I don''t
have a problem.
So, once again, this is not particular to a single device. It affects all
devices if the condition mentioned above is true.
Thanks.
-Anil
________________________________
From: theubaz@gmail.com [mailto:theubaz@gmail.com] On Behalf Of John Sherwood
Sent: Thursday, May 31, 2012 9:16 PM
To: Anil Rao
Cc: xen-users@lists.xen.org
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
If you''ve never seen this issue before and can''t reproduce it
with any other PCI devices, I''d suggest trying the devel mailing list,
but I''d first collect whatever logs you can that are salient to the
issue beforehand. I''ve never had issues with PCI hot plug that were
limited to a single device though, so I''m certainly drawing a blank
here on what it could be.
On Thu, May 31, 2012 at 8:06 PM, Anil Rao
<anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote:
Since I am new to the xen-users list, I was wondering if my question is more
appropriate for the xen-devel list. Any thoughts?
Thanks.
-Anil
________________________________
From:
xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>
[mailto:xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>]
On Behalf Of Anil Rao
Sent: Thursday, May 31, 2012 11:25 AM
To: cyberhawk001@gmail.com<mailto:cyberhawk001@gmail.com>;
xen-users@lists.xen.org<mailto:xen-users@lists.xen.org>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
This is a proprietary FPGA device. Each control blade in our router chassis has
one of these FPGAs and its primary function is to control certain dedicated
hardware components on the blade that are responsible for establishing
connectivity with the high-speed inter-blade fabric on the router. I am sorry,
but I cannot divulge additional details regarding the FPGA.
Here is a more detailed description of my actual usage scenario.
1) The FPGA image must be downloaded onto the device before it can be enabled
for use.
2) The VM in question is responsible for performing step (1), which means that
the FPGA is not visible to Xen/Dom-0 when
the VM is powered on.
3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen
pci-hotplug to assign the device to the VM.
This operation completes successfully (as reported by the appropriate xl
command).
4) However, the guest OS inside the VM cannot see the FPGA device after step
(3).
5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the
FPGA device (and everything then proceeds normally).
The troublesome part here is that I need to shutdown and restart the VM. I would
like to avoid this and rely on Xen pci-hotplug to make the device visible to the
guest OS inside the VM.
Thanks.
-Anil
________________________________
From:
xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>
[mailto:xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>]
On Behalf Of cyberhawk001@gmail.com<mailto:cyberhawk001@gmail.com>
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@lists.xen.org<mailto:xen-users@lists.xen.org>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Just out of curiosity, what kind of FPGA device is this? Is it an FPGA
development board or something already assembled and functioning that contains
an FPGA device(s)?
Well, just being curious... :)
Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI
devices to the VM. The only one that doesn''t work is the FPGA device.
Even for this one, the only time it doesn''t work, is if the VM was
powered-on prior to the FPGA device being brought on-line on the host. If the
FPGA was present before the VM was powered-on everything works as expected.
It somehow seems like new devices that are introduced in the host, after a VM
has been started on it, cannot be hotplugged into the VM.
Thanks.
-Anil
________________________________
From: theubaz@gmail.com<mailto:theubaz@gmail.com>
[mailto:theubaz@gmail.com] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Are you sure the guest OS has PCI hotplug support?
On Wed, May 30, 2012 at 10:14 PM, Anil Rao
<anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote:
Hi,
I am trying to assign a FPGA (which is a PCI device) to a running VM. The
interesting thing here is that the FPGA is not present when the VM was
powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line
on the host by downloading its image to the device.
So far my attempts at using Xen PCI-passthrough have not been successful.
Although Xen reports that the pass-through operation was successful, which I
have verified using the appropriate xl command, I am unable to see the FPGA from
within the guest OS.
I have tried both manual rescanning of the PCI bus inside the guest as well as
making use of the pciehp driver. The only way I can (reliably) see the FPGA from
within the guest, is to shutdown and restart the VM and then redo the hotplug
operation. I want to avoid a VM reboot and was wondering if this is at all
possible and if so what do I need to do to realize it.
[The virtualization host is based on the default Xen setup that comes with SLES
11 SP2]
Thanks for any help on this matter.
-Anil
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org<mailto:Xen-users@lists.xen.org>
http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
After poking around some more on my host, I noticed the following differences in
the output of qemu-dm, which are shown below (extracted from
/var/log/xen/qemu-dm-VMName.log).
Case 1: VM was powered on ''before'' PCI device (FPGA) was
brought on-line
-------------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Error: couldn''t locate device in libpci
structures
Case 2: VM was powered on ''after'' PCI device (FPGA) was
brought on-line
----------------------------------------------------------------------------------------------------------
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can''t open file /dev/xen/pci_iomul:
No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6000000)
pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6100000)
pt_msi_setup: msi mapped with pirq 84 pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
So clearly, qemu-dm reported an error for case (1) and reported success for case
(2).
The interesting thing is why did the Xen xm pci-attach command report success in
both cases. Also, from Xen''s perspective it appears that the
afore-mentioned PCI device is assigned to the VM in both cases (as verified by
using the xm commands to get the list of remaining assignable PCI devices and
the list of PCI devices attached to the VM).
Something seems wrong here.
Any thoughts?
-Anil
________________________________
From:
xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>
[mailto:xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>]
On Behalf Of Anil Rao
Sent: Thursday, May 31, 2012 11:25 AM
To: cyberhawk001@gmail.com<mailto:cyberhawk001@gmail.com>;
xen-users@lists.xen.org<mailto:xen-users@lists.xen.org>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
This is a proprietary FPGA device. Each control blade in our router chassis has
one of these FPGAs and its primary function is to control certain dedicated
hardware components on the blade that are responsible for establishing
connectivity with the high-speed inter-blade fabric on the router. I am sorry,
but I cannot divulge additional details regarding the FPGA.
Here is a more detailed description of my actual usage scenario.
1) The FPGA image must be downloaded onto the device before it can be enabled
for use.
2) The VM in question is responsible for performing step (1), which means that
the FPGA is not visible to Xen/Dom-0 when
the VM is powered on.
3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen
pci-hotplug to assign the device to the VM.
This operation completes successfully (as reported by the appropriate xl
command).
4) However, the guest OS inside the VM cannot see the FPGA device after step
(3).
5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the
FPGA device (and everything then proceeds normally).
The troublesome part here is that I need to shutdown and restart the VM. I would
like to avoid this and rely on Xen pci-hotplug to make the device visible to the
guest OS inside the VM.
Thanks.
-Anil
________________________________
From:
xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>
[mailto:xen-users-bounces@lists.xen.org<mailto:xen-users-bounces@lists.xen.org>]
On Behalf Of cyberhawk001@gmail.com<mailto:cyberhawk001@gmail.com>
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@lists.xen.org<mailto:xen-users@lists.xen.org>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Just out of curiosity, what kind of FPGA device is this? Is it an FPGA
development board or something already assembled and functioning that contains
an FPGA device(s)?
Well, just being curious... :)
Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI
devices to the VM. The only one that doesn''t work is the FPGA device.
Even for this one, the only time it doesn''t work, is if the VM was
powered-on prior to the FPGA device being brought on-line on the host. If the
FPGA was present before the VM was powered-on everything works as expected.
It somehow seems like new devices that are introduced in the host, after a VM
has been started on it, cannot be hotplugged into the VM.
Thanks.
-Anil
________________________________
From: theubaz@gmail.com<mailto:theubaz@gmail.com>
[mailto:theubaz@gmail.com] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
Subject: Re: [Xen-users] Hotplugging FPGA to running VM
Are you sure the guest OS has PCI hotplug support?
On Wed, May 30, 2012 at 10:14 PM, Anil Rao
<anil.rao@ericsson.com<mailto:anil.rao@ericsson.com>> wrote:
Hi,
I am trying to assign a FPGA (which is a PCI device) to a running VM. The
interesting thing here is that the FPGA is not present when the VM was
powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line
on the host by downloading its image to the device.
So far my attempts at using Xen PCI-passthrough have not been successful.
Although Xen reports that the pass-through operation was successful, which I
have verified using the appropriate xl command, I am unable to see the FPGA from
within the guest OS.
I have tried both manual rescanning of the PCI bus inside the guest as well as
making use of the pciehp driver. The only way I can (reliably) see the FPGA from
within the guest, is to shutdown and restart the VM and then redo the hotplug
operation. I want to avoid a VM reboot and was wondering if this is at all
possible and if so what do I need to do to realize it.
[The virtualization host is based on the default Xen setup that comes with SLES
11 SP2]
Thanks for any help on this matter.
-Anil
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org<mailto:Xen-users@lists.xen.org>
http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
It''s quite possible that the error simply isn''t propagating up to Xen during the pci-attach command (and that would be my guess, given that qemu-dm is noticing the error and xen isn''t). It would appear though that you''ve found "why" it''s failing, although who knows long it might take to figure out the actual error in register_real_device. This is probably where you''ll need to contact the developer list to make any progress. On Mon, Jun 4, 2012 at 4:34 PM, Anil Rao <anil.rao@ericsson.com> wrote:> ** > After poking around some more on my host, I noticed the following > differences in the output of qemu-dm, which are shown below (extracted from > /var/log/xen/qemu-dm-VMName.log). > > > Case 1: VM was powered on ''before'' PCI device (FPGA) was brought on-line > > ------------------------------------------------------------------------------------------------------------- > dm-command: hot insert pass-through pci dev > register_real_device: Assigning real physical device 01:00.0 ... > register_real_device: Error: couldn''t locate device in libpci structures > > > Case 2: VM was powered on ''after'' PCI device (FPGA) was brought on-line > > ---------------------------------------------------------------------------------------------------------- > dm-command: hot insert pass-through pci dev > register_real_device: Assigning real physical device 01:00.0 ... > register_real_device: Enable MSI translation via per device option > register_real_device: Disable power management > pt_iomul_init: Error: pt_iomul_init can''t open file /dev/xen/pci_iomul: No > such file or directory: 0x1:0x0.0x0 > pt_register_regions: IO region registered (size=0x00100000 > base_addr=0xc6000000) > pt_register_regions: IO region registered (size=0x00100000 > base_addr=0xc6100000) > pt_msi_setup: msi mapped with pirq 84 pci_intx: intx=1 > register_real_device: Real physical device 01:00.0 registered successfuly! > > So clearly, qemu-dm reported an error for case (1) and reported success > for case (2). > > The interesting thing is why did the Xen xm pci-attach command report > success in both cases. Also, from Xen''s perspective it appears that the > afore-mentioned PCI device is assigned to the VM in both cases (as verified > by using the xm commands to get the list of remaining assignable PCI > devices and the list of PCI devices attached to the VM). > > Something seems wrong here. > > Any thoughts? > > -Anil > > ------------------------------ > *From:* xen-users-bounces@lists.xen.org [mailto: > xen-users-bounces@lists.xen.org] *On Behalf Of *Anil Rao > *Sent:* Thursday, May 31, 2012 11:25 AM > *To:* cyberhawk001@gmail.com; xen-users@lists.xen.org > >> >> *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM >> >> This is a proprietary FPGA device. Each control blade in our router >> chassis has one of these FPGAs and its primary function is to control >> certain dedicated hardware components on the blade that are responsible for >> establishing connectivity with the high-speed inter-blade fabric on the >> router. I am sorry, but I cannot divulge additional details regarding the >> FPGA. >> >> Here is a more detailed description of my actual usage scenario. >> >> 1) The FPGA image must be downloaded onto the device before it can be >> enabled for use. >> 2) The VM in question is responsible for performing step (1), which means >> that the FPGA is not visible to Xen/Dom-0 when >> the VM is powered on. >> 3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen >> pci-hotplug to assign the device to the VM. >> This operation completes successfully (as reported by the appropriate >> xl command). >> 4) However, the guest OS inside the VM cannot see the FPGA device after >> step (3). >> 5) If I shutdown the VM and re-start it, the guest OS inside the VM sees >> the FPGA device (and everything then proceeds normally). >> >> The troublesome part here is that I need to shutdown and restart the VM. >> I would like to avoid this and rely on Xen pci-hotplug to make the device >> visible to the guest OS inside the VM. >> >> Thanks. >> -Anil >> >> ------------------------------ >> *From:* xen-users-bounces@lists.xen.org [mailto: >> xen-users-bounces@lists.xen.org] *On Behalf Of *cyberhawk001@gmail.com >> *Sent:* Thursday, May 31, 2012 10:49 AM >> *To:* xen-users@lists.xen.org >> *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM >> >> Just out of curiosity, what kind of *FPGA device* is this? Is it an FPGA >> development board or something already assembled and functioning that >> contains an FPGA device(s)? >> >> Well, just being curious... :) >> >> >> >> Yes, the guest OS has PCI hotplug support. I am able to hotplug a >> variety of PCI devices to the VM. The only one that doesn''t work is the >> FPGA device. Even for this one, the only time it doesn''t work, is if the VM >> was powered-on prior to the FPGA device being brought on-line on the host. >> If the FPGA was present before the VM was powered-on everything works as >> expected. >> >> It somehow seems like new devices that are introduced in the host, after >> a VM has been started on it, cannot be hotplugged into the VM. >> >> Thanks. >> -Anil >> >> >> ------------------------------ >> *From:* theubaz@gmail.com [mailto:theubaz@gmail.com <theubaz@gmail.com>] >> *On Behalf Of *John Sherwood >> *Sent:* Wednesday, May 30, 2012 7:20 PM >> *To:* Anil Rao >> *Cc:* xen-users@lists.xensource.com >> *Subject:* Re: [Xen-users] Hotplugging FPGA to running VM >> >> Are you sure the guest OS has PCI hotplug support? >> >> On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@ericsson.com> wrote: >> >>> Hi, >>> >>> I am trying to assign a FPGA (which is a PCI device) to a running VM. >>> The interesting thing here is that the FPGA is not present when the VM was >>> powered-on. I.e. sometime after the VM was started, the FPGA was brought >>> on-line on the host by downloading its image to the device. >>> >>> So far my attempts at using Xen PCI-passthrough have not been >>> successful. Although Xen reports that the pass-through operation was >>> successful, which I have verified using the appropriate xl command, I am >>> unable to see the FPGA from within the guest OS. >>> >>> I have tried both manual rescanning of the PCI bus inside the guest as >>> well as making use of the pciehp driver. The only way I can (reliably) see >>> the FPGA from within the guest, is to shutdown and restart the VM and then >>> redo the hotplug operation. I want to avoid a VM reboot and was wondering >>> if this is at all possible and if so what do I need to do to realize it. >>> >>> [The virtualization host is based on the default Xen setup that comes >>> with SLES 11 SP2] >>> >>> Thanks for any help on this matter. >>> >>> -Anil >>> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xen.org >> http://lists.xen.org/xen-users >> >> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users