Hi all, I am having problems getting SR-IOV functions to work in my HVMs. My hardware has VT-d support, and pci passthrough works fine for physical functions. In a nutshell here is the current state: Dom0 : NICs are 2-port 82576. I can get full network access using either PF or VF interfaces. HVM : PCI passthrough of physical functions work -- full network access HVM : PCI passthrough of virtual functions fail -- can send pkts but cannot receive. The best lead I have right now is evident from the qemu logs. when PF (04:00.0) is assigned to HVM: pt_msix_init: get MSI-X table bar base fafbc000 pt_msix_init: table_off = 0, total_entries = 10 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 pt_msi_setup: msi mapped with pirq 37 pci_intx: intx=1 register_real_device: Real physical device 04:00.0 registered successfuly! IRQ type = MSI-INTx when VF (04:10.2) is assigned to HVM: pt_msix_init: get MSI-X table bar base fae24000 pt_msix_init: table_off = 0, total_entries = 3 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7fc918846000 register_real_device: Real physical device 04:10.2 registered successfuly! IRQ type = INTx VFs don''t seem to be using MSI/MSI-X interupts. Does this indicate a problem? Can anyone give some insight on this? thanks - david _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rose, Gregory V
2011-Mar-01 20:50 UTC
RE: [Xen-devel] SR-IOV problems - HVM cannot access network
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of David White > Sent: Tuesday, March 01, 2011 11:32 AM > To: xen-devel@lists.xensource.com > Subject: [Xen-devel] SR-IOV problems - HVM cannot access network > > Hi all, > > I am having problems getting SR-IOV functions to work in my HVMs. My > hardware has VT-d support, and pci passthrough works fine for physical > functions. In a nutshell here is the current state: > > Dom0 : NICs are 2-port 82576. I can get full network access using > either PF or VF interfaces. > HVM : PCI passthrough of physical functions work -- full network access > HVM : PCI passthrough of virtual functions fail -- can send pkts but > cannot receive. > > The best lead I have right now is evident from the qemu logs. > > when PF (04:00.0) is assigned to HVM: > > pt_msix_init: get MSI-X table bar base fafbc000 > pt_msix_init: table_off = 0, total_entries = 10 > pt_msix_init: errno = 2 > pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 > pt_msi_setup: msi mapped with pirq 37 > pci_intx: intx=1 > register_real_device: Real physical device 04:00.0 registered successfuly! > IRQ type = MSI-INTx > > when VF (04:10.2) is assigned to HVM: > > pt_msix_init: get MSI-X table bar base fae24000 > pt_msix_init: table_off = 0, total_entries = 3 > pt_msix_init: errno = 2 > pt_msix_init: mapping physical MSI-X table to 7fc918846000 > register_real_device: Real physical device 04:10.2 registered successfuly! > IRQ type = INTx > > VFs don''t seem to be using MSI/MSI-X interupts. Does this indicate a > problem?Yes, this is absolutely a problem. 82576 virtual functions require MSI-X interrupt support to function properly. You didn''t mention what your guest OS is but the guest OS must support MSI-X interrupts. Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason. If that happens then the VF will not function correctly. - Greg Rose LAN Access Division Intel Corp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David White
2011-Mar-01 22:04 UTC
Re: [Xen-devel] SR-IOV problems - HVM cannot access network
On 03/01/2011 12:50 PM, Rose, Gregory V wrote:>> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of David White >> Sent: Tuesday, March 01, 2011 11:32 AM >> To: xen-devel@lists.xensource.com >> Subject: [Xen-devel] SR-IOV problems - HVM cannot access network >> >> Hi all, >> >> I am having problems getting SR-IOV functions to work in my HVMs. My >> hardware has VT-d support, and pci passthrough works fine for physical >> functions. In a nutshell here is the current state: >> >> Dom0 : NICs are 2-port 82576. I can get full network access using >> either PF or VF interfaces. >> HVM : PCI passthrough of physical functions work -- full network access >> HVM : PCI passthrough of virtual functions fail -- can send pkts but >> cannot receive. >> >> The best lead I have right now is evident from the qemu logs. >> >> when PF (04:00.0) is assigned to HVM: >> >> pt_msix_init: get MSI-X table bar base fafbc000 >> pt_msix_init: table_off = 0, total_entries = 10 >> pt_msix_init: errno = 2 >> pt_msix_init: mapping physical MSI-X table to 7f23a03d5000 >> pt_msi_setup: msi mapped with pirq 37 >> pci_intx: intx=1 >> register_real_device: Real physical device 04:00.0 registered successfuly! >> IRQ type = MSI-INTx >> >> when VF (04:10.2) is assigned to HVM: >> >> pt_msix_init: get MSI-X table bar base fae24000 >> pt_msix_init: table_off = 0, total_entries = 3 >> pt_msix_init: errno = 2 >> pt_msix_init: mapping physical MSI-X table to 7fc918846000 >> register_real_device: Real physical device 04:10.2 registered successfuly! >> IRQ type = INTx >> >> VFs don''t seem to be using MSI/MSI-X interupts. Does this indicate a >> problem? > Yes, this is absolutely a problem. 82576 virtual functions require MSI-X interrupt support to function properly. You didn''t mention what your guest OS is but the guest OS must support MSI-X interrupts. Even if it does have MSI-X support the attempt to allocate the vectors may fail for some reason. If that happens then the VF will not function correctly. > > - Greg Rose > LAN Access Division > Intel Corp. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-develThe guest (Ubuntu Maverick 64-bit) supports MSI-X, as is indicated by the fact that PF passthrough works fine (am I interpreting the above output correctly, for the PF/HVM case?). The only thing different between the two above cases is that the first one is assigned a PF, the second on a VF. Is there something in the dom0 igb/igbvf drivers that affect the MSI-X capabilities in the HVM? igb/igbvf drivers on dom0 are from Intel site: root@dom0:~# modinfo igb | grep ^version: version: 2.4.12 root@dom0:~# modinfo igbvf | grep ^version: version: 1.0.7 The guest driver does not seem to be the source of this difference, since these qemu logs are logged before HVM grub menu appears (and hence before guest kernel is loaded) Why would a physical function invoke MSI-X but not a virtual function? Is it something that pciback is doing? When I bind a PF to pciback, the xen-pciback driver shows: [ 1390.090693] pciback 0000:04:00.1: seizing device [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 [ 1390.101021] Already setup the GSI :17 [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17 [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled but when I bind a VF to pciback, the driver shows: [ 1439.411763] pciback 0000:04:10.0: seizing device [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) (note no INT or GSI messages) could pciback be the source of the HVM INTx problem? -David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rose, Gregory V
2011-Mar-01 22:41 UTC
RE: [Xen-devel] SR-IOV problems - HVM cannot access network
> -----Original Message----- > From: David White [mailto:dwhite@speakeasy.net] > Sent: Tuesday, March 01, 2011 2:05 PM > To: Rose, Gregory V > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network >[snip}> > The guest driver does not seem to be the source of this difference, > since these qemu logs are logged before HVM grub menu appears (and hence > before guest kernel is loaded) > > Why would a physical function invoke MSI-X but not a virtual function?A lot of reasons actually :-). SR-IOV is a complete platform solution and feature. It requires full support from all PCIe devices from the root complex to the endpoint device as well as proper BIOS programming and OS support. I don''t know that Ubuntu actually supports SR-IOV and I don''t know which platform you''re using and whether it supports SR-IOV.> Is it something that pciback is doing? > > When I bind a PF to pciback, the xen-pciback driver shows: > [ 1390.090693] pciback 0000:04:00.1: seizing device > [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 > [ 1390.101021] Already setup the GSI :17 > [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> > IRQ 17 > [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled > > but when I bind a VF to pciback, the driver shows: > [ 1439.411763] pciback 0000:04:10.0: seizing device > [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) > > (note no INT or GSI messages) > > could pciback be the source of the HVM INTx problem?Well it''s certainly possible but another good possibility is that the machine doesn''t actually implement the SR-IOV feature correctly in the BIOS. What machine are you using? Is there a BIOS setting to enable SR-IOV and if so are you sure you''ve enabled it? Also, would it be possible to get a register dump of the VF device from both the Dom0 host domain and the DomU guest domain using the Intel ethregs utility? You can get that from source forge. - Greg _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David White
2011-Mar-02 02:41 UTC
Re: [Xen-devel] SR-IOV problems - HVM cannot access network
On 03/01/2011 02:41 PM, Rose, Gregory V wrote:>> -----Original Message----- >> From: David White [mailto:dwhite@speakeasy.net] >> Sent: Tuesday, March 01, 2011 2:05 PM >> To: Rose, Gregory V >> Cc:xen-devel@lists.xensource.com >> Subject: Re: [Xen-devel] SR-IOV problems - HVM cannot access network >> > [snip} > >> The guest driver does not seem to be the source of this difference, >> since these qemu logs are logged before HVM grub menu appears (and hence >> before guest kernel is loaded) >> >> Why would a physical function invoke MSI-X but not a virtual function? > A lot of reasons actually :-). SR-IOV is a complete platform solution and > feature. It requires full support from all PCIe devices from the root complex > to the endpoint device as well as proper BIOS programming and OS support. > > I don''t know that Ubuntu actually supports SR-IOV and I don''t know which > platform you''re using and whether it supports SR-IOV. > >> Is it something that pciback is doing? >> >> When I bind a PF to pciback, the xen-pciback driver shows: >> [ 1390.090693] pciback 0000:04:00.1: seizing device >> [ 1390.095417] xen_allocate_pirq: returning irq 17 for gsi 17 >> [ 1390.101021] Already setup the GSI :17 >> [ 1390.104717] pciback 0000:04:00.1: PCI INT B -> GSI 17 (level, low) -> >> IRQ 17 >> [ 1390.111807] pciback 0000:04:00.1: PCI INT B disabled >> >> but when I bind a VF to pciback, the driver shows: >> [ 1439.411763] pciback 0000:04:10.0: seizing device >> [ 1439.416462] pciback 0000:04:10.0: enabling device (0000 -> 0002) >> >> (note no INT or GSI messages) >> >> could pciback be the source of the HVM INTx problem? > Well it''s certainly possible but another good possibility is that the machine > doesn''t actually implement the SR-IOV feature correctly in the BIOS. What > machine are you using? Is there a BIOS setting to enable SR-IOV and if so are > you sure you''ve enabled it? > > Also, would it be possible to get a register dump of the VF device from both > the Dom0 host domain and the DomU guest domain using the Intel ethregs > utility? You can get that from source forge. > > - Greg >Thanks for your help so far. Ubuntu Maverick is only running in the guest HVM domains. I have two boards I am testing with, both with Intel 5520 chipset. I''m using two 82576 nics from two different vendors, one attached to either board. Both systems have AMI bios. One system (call it systemA) has bios options for both VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) has bios options for VT-d and "Coherency support" (which seemed to help me enumerate the VFs), and both are enabled (no mention of I/OAT). SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. SystemB runs Fedora14 with the well-known "jeremy" kernel (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), and xen-4.0.1 hypervisor built from source (unaltered). Both systems are experiencing the same behavior. attached are dom0 and hvm ethregs output for the VF using ''-o -m'' flags. This one was taken from the Debian host ("systemA") -dave _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Mar-02 11:06 UTC
Re: [Xen-devel] SR-IOV problems - HVM cannot access network
On Wed, 2 Mar 2011, David White wrote:> Thanks for your help so far. > > Ubuntu Maverick is only running in the guest HVM domains. > > I have two boards I am testing with, both with Intel 5520 chipset. I''m using > two 82576 nics from two different vendors, one attached to either board. Both > systems have AMI bios. One system (call it systemA) has bios options for both > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > has bios options for VT-d and "Coherency support" (which seemed to help me > enumerate the VFs), and both are enabled (no mention of I/OAT). > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > SystemB runs Fedora14 with the well-known "jeremy" kernel > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > and xen-4.0.1 hypervisor built from source (unaltered). > > Both systems are experiencing the same behavior. > > attached are dom0 and hvm ethregs output for the VF using ''-o -m'' > flags. This one was taken from the Debian host ("systemA")could you please try adding acpi=0 in your VM config file? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David white
2011-Mar-02 20:55 UTC
Re: [Xen-devel] SR-IOV problems - HVM cannot access network
On Wed, 2 Mar 2011, Stefano Stabellini wrote:> On Wed, 2 Mar 2011, David White wrote: > > Thanks for your help so far. > > > > Ubuntu Maverick is only running in the guest HVM domains. > > > > I have two boards I am testing with, both with Intel 5520 chipset. I''m using > > two 82576 nics from two different vendors, one attached to either board. Both > > systems have AMI bios. One system (call it systemA) has bios options for both > > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > > has bios options for VT-d and "Coherency support" (which seemed to help me > > enumerate the VFs), and both are enabled (no mention of I/OAT). > > > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > > SystemB runs Fedora14 with the well-known "jeremy" kernel > > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > > and xen-4.0.1 hypervisor built from source (unaltered). > > > > Both systems are experiencing the same behavior. > > > > attached are dom0 and hvm ethregs output for the VF using ''-o -m'' > > flags. This one was taken from the Debian host ("systemA") > > could you please try adding > > acpi=0 > > in your VM config file? >Hm, ok that seemed to do the trick, as far as getting the VFs up on the network. Is this by design or is it a workaround? Must I sacrifice acpi in my guest to enable sr-iov? Does this come with a cost to I/O performance, since the guest isn''t using MSI? -David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Mar-02 21:58 UTC
Re: [Xen-devel] SR-IOV problems - HVM cannot access network
On Wed, 2 Mar 2011, David white wrote:> > On Wed, 2 Mar 2011, Stefano Stabellini wrote: > > > On Wed, 2 Mar 2011, David White wrote: > > > Thanks for your help so far. > > > > > > Ubuntu Maverick is only running in the guest HVM domains. > > > > > > I have two boards I am testing with, both with Intel 5520 chipset. I''m using > > > two 82576 nics from two different vendors, one attached to either board. Both > > > systems have AMI bios. One system (call it systemA) has bios options for both > > > VT-d and SR-IOV, and both are enabled, as is I/OAT. The other system (systemB) > > > has bios options for VT-d and "Coherency support" (which seemed to help me > > > enumerate the VFs), and both are enabled (no mention of I/OAT). > > > > > > SystemA runs Debian Squeeze and the xen kernel, vmm and tools from that distro. > > > SystemB runs Fedora14 with the well-known "jeremy" kernel > > > (http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x), > > > and xen-4.0.1 hypervisor built from source (unaltered). > > > > > > Both systems are experiencing the same behavior. > > > > > > attached are dom0 and hvm ethregs output for the VF using ''-o -m'' > > > flags. This one was taken from the Debian host ("systemA") > > > > could you please try adding > > > > acpi=0 > > > > in your VM config file? > > > > Hm, ok that seemed to do the trick, as far as getting the VFs up on the network. >Thank you very much for testing this, it proves that the recent problems with VF passthrough and HVM Linux guests are the same issue reported here: http://marc.info/?l=xen-devel&m=129847075105008&w=2 and here: http://marc.info/?l=xen-devel&m=129891005422870&w=2> Is this by design or is it a workaround? Must I sacrifice acpi in my guest to > enable sr-iov? Does this come with a cost to I/O performance, since the guest > isn''t using MSI?It is just a workaround. It shouldn''t cost you any performance at all, but you won''t be able to hot-plug or hot-unplug PCI devices to your guests. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel