Hi, I am trying to setup 82576 VF passthrough to pv domU non Vt-d machine . My dom0 is http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/ kernel on FC11 and 3.4.1 Xen. My domU is also http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/kernel on Ubuntu junty. Passthrough of PF device (igb driver) is successful in my setup. I could see the PF BDF on domU, and the igb driver gets loaded on domU. But, when I do passthrough of VF device, lspci output on DomU doesn''t show the VF device.(CONFIG_XEN_PCIDEV_FRONTEND=y) The steps I follow on Dom0: #rmmod igb #modprobe igb max_vfs=1 #modprobe pciback #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/new_slot #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/bind On domU config file has pci = ['' VF-BDF''] . The domU starts without any errors. #xm pci-list domU-ID shows the VF BDF Am i missing some thing of VF Passthrough? Please give me pointers to find/debug the issue. Regards, -Satish _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote:> Hi, > > I am trying to setup 82576 VF passthrough to pv domU non Vt-d machine . My > dom0 is http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/ kernel on FC11 and > 3.4.1 Xen. My domU is also > http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/kernel on Ubuntu > junty. > > > Passthrough of PF device (igb driver) is successful in my setup. I could see > the PF BDF on domU, and the igb driver gets loaded on domU. > > But, when I do passthrough of VF device, lspci output on DomU doesn''t show > the VF device.(CONFIG_XEN_PCIDEV_FRONTEND=y) > > The steps I follow on Dom0: > #rmmod igb > #modprobe igb max_vfs=1 > #modprobe pciback > #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/new_slot > #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/bind > On domU config file has pci = ['' VF-BDF''] . > > The domU starts without any errors. > > #xm pci-list domU-ID > shows the VF BDF > > Am i missing some thing of VF Passthrough? Please give me pointers to > find/debug the issue.Hi Satish, I''m not sure why your setup isn''t working but I''m not aware of any tricks that are required. Though I should point out that the linux-2.6.18-xen-igb kernel does not include an igbvf driver so you will need to run a different kernel in your domUs - I used 2.6.31 for testing purposes. On the off chance that it is useful to you here are some details of a working system that I have. Hardware -------- Machine: HP dc7800 MT: [japanese] http://h50146.www5.hp.com/products/desktops/old/dc7800mt/e8300_10_160w_xpv.html Chipset: Q35 82576 PCI Card: E1G42ET (dual port RJ-45 copper) Dom0 ---- The dom0 kernel config is attached as config-2.6.18-xen-igb. I ran my test using the following revision of http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/ changeset: 1123:f4deded581e6 user: Simon Horman <horms@verge.net.au> date: Mon Aug 31 09:54:00 2009 +1000 summary: Merge with http://xenbits.xensource.com/linux-2.6.18-xen.hg The hypervisor and xend comes from the following revision of http://xenbits.xensource.com/staging/xen-unstable.hg, which should also exist in http://xenbits.xensource.com/xen-unstable.hg changeset: 20010:68e8b8379244 user: Keir Fraser <keir.fraser@citrix.com> date: Sun Aug 02 13:43:15 2009 +0100 summary: Add a single trigger for all diagnostic keyhandlers qemu-xen is the following revision of git://xenbits.xensource.com/staging/qemu-xen-unstable.git/, which should also exist in git://xenbits.xensource.com/qemu-xen-unstable.git/ commit a42b67ed27a9744a2492c8ffe5c26935bd19ee27 Author: Ian Jackson <ian.jackson@eu.citrix.com> Date: Tue Jul 21 16:08:24 2009 +0100 use XC_PAGE_SIZE in xenfbfront (fix stubdom build on NetBSD) Apropos of a report from Christoph Egger about misuse of PAGE_SIZE in xenfbfront; change references to XC_PAGE_SIZE which is correct in our context. This fix suggested by Stefano Stabillini. Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> The grub snippet for dom0 is as follows. pci=assign-busses is needed because the BIOS on my machine does not appear to fully support VT-d and without it there is nowhere for the VFs to go. reassign_resources and reassigndev=00:1d.7 are spurious. title Xen root (hd0,0) kernel /xen com1=115200,8n1,0x3f8,0 console=com1 loglvl=all guest_loglvl=all vtd=1 iommu dom0_mem=1G module /vmlinuz-2.6.18.8 root=/dev/mapper/sam-root_x86_64 console=ttyS0 loglevel=8 ro pci=assign-busses reassign_resources reassigndev=00:1d.7 module /initrd.img-2.6.18.8 The igb module was inserted into the kernel using modprobe igb max_vfs=4 The output of ''dmesg | grep igb'' in domU is: igb 0000:01:00.0: pci_enable_pcie_error_reporting failed 0xffffffea igb 0000:01:00.0: 4 vfs allocated igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection igb 0000:01:00.0: eth3: (PCIe:2.5Gb/s:Width x4) ffff8800389841a8M igb 0000:01:00.0: eth3: PBA No: e43709-003 igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) igb 0000:01:00.1: pci_enable_pcie_error_reporting failed 0xffffffea igb 0000:01:00.1: 4 vfs allocated igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection igb 0000:01:00.1: eth3: (PCIe:2.5Gb/s:Width x4) ffff8800388101a8M igb 0000:01:00.1: eth3: PBA No: e43709-003 igb 0000:01:00.1: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) DomU ---- The domU kernel config is attached as config-2.6.31, and I have verified that this works with 2.6.31. The domU grub snippet is: title Linux root (hd0,0) kernel /boot/vmlinuz-2.6.31 root=/dev/hda1 ro loglevel=8 console=uart,io,0x3f8,115200n8 The domU domain config is attached as hvm and I invoke the domain using: xm create hvm pci=''02:10.0'' Where 02:10.0 is one of the vf devices. The output of lspci in dom0 is: 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Class ff80: XenSource, Inc. Xen Platform Device (rev 01) 00:04.0 Ethernet controller: Intel Corporation Device 10ca (rev 01) The output of ''dmesg | grep igb'' in domU is: [ 3.765661] igbvf 0000:00:04.0: irq 48 for MSI/MSI-X [ 3.768937] igbvf 0000:00:04.0: irq 49 for MSI/MSI-X [ 3.772200] igbvf 0000:00:04.0: irq 50 for MSI/MSI-X [ 3.808717] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function [ 3.813071] igbvf 0000:00:04.0: Address: da:3a:32:ee:00:30 [ 3.816918] igbvf 0000:00:04.0: MAC: 1 [ 7.544165] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex Other ----- Note that if you want to see a link (assuming that everything else works) on the igbvf device inside the domain, you''ll need to bring up the link on the PF in dom0. ''ifconfig ethX up'' should be sufficient. The dom0 and domU user-spaces are basically Debian Lenny, though I doubt this has any influence on the results. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of >Simon Horman >Sent: Wednesday, September 23, 2009 4:25 PM >To: Satish Chowdhury >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] 82576 VF passthrough > >On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote: >> Hi, >> >> I am trying to setup 82576 VF passthrough to pv domU non >Vt-d machineYou can''t run SR-IOV VF devices in VM guests on a non Vt-d machine. VT-d is required for VF passthrough to a guest. - Greg LAD Intel Corp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Greg, I am using Vt-d machine. Sorry for the confusion. My Hardware is HP dc7800p Chipset: Q35, with Intel Core 2 Duo E8400 processor. NIC Card is 82576 E1G42ET (dual port RJ-45 copper). Is VF passthrough can be done only to HVM guest? Is VF passthrough to paravirtal guest is supported? Regards, -Satish On Fri, Sep 25, 2009 at 10:36 AM, Rose, Gregory V <gregory.v.rose@intel.com>wrote:> > >-----Original Message----- > >From: xen-devel-bounces@lists.xensource.com > >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > >Simon Horman > >Sent: Wednesday, September 23, 2009 4:25 PM > >To: Satish Chowdhury > >Cc: xen-devel@lists.xensource.com > >Subject: Re: [Xen-devel] 82576 VF passthrough > > > >On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote: > >> Hi, > >> > >> I am trying to setup 82576 VF passthrough to pv domU non > >Vt-d machine > > You can''t run SR-IOV VF devices in VM guests on a non Vt-d machine. VT-d > is required for VF passthrough to a guest. > > - Greg > LAD > Intel Corp. > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''ve only ever used HVMs. I can''t say whether VF passthrough to a PV guest works or not but I can''t think of any reason it wouldn''t. Can you pass other devices through to this PV guest? If so then a VF should work also. Generally any guest that you can a physical device through will also work with a VF device. Next thing to consider is whether your BIOS supports SR-IOV. There are a lot of machines that have the necessary HW to support SR-IOV but the BIOS does not properly detect SR-IOV capable devices and program the upstream switches and bridges properly to make sure the I/O aperture is large enough to accommodate the extra VF devices when they are enabled by the driver. There are very few BIOS vendors shipping SR-IOV capable BIOS at the moment. If your machine does not have the right BIOS then there is a work around in the Xen 3.4 release that you can use. Or at least I think it''s still there. pci=assign-mmio=0000:01 Try adding that to your kernel command line but substitute the domain/bus of your device. For instance if your Kawela is on bus 4 then you would change the command to pci=assign-mmio=0000:04. Hope that helps. - Greg ________________________________ From: Satish Chowdhury [mailto:satish.chowdhury@oneconvergence.com] Sent: Thursday, September 24, 2009 10:32 PM To: Rose, Gregory V Cc: Simon Horman; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] 82576 VF passthrough Hi Greg, I am using Vt-d machine. Sorry for the confusion. My Hardware is HP dc7800p Chipset: Q35, with Intel Core 2 Duo E8400 processor. NIC Card is 82576 E1G42ET (dual port RJ-45 copper). Is VF passthrough can be done only to HVM guest? Is VF passthrough to paravirtal guest is supported? Regards, -Satish On Fri, Sep 25, 2009 at 10:36 AM, Rose, Gregory V <gregory.v.rose@intel.com<mailto:gregory.v.rose@intel.com>> wrote:>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com> >[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>] On Behalf Of >Simon Horman >Sent: Wednesday, September 23, 2009 4:25 PM >To: Satish Chowdhury >Cc: xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com> >Subject: Re: [Xen-devel] 82576 VF passthrough > >On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote: >> Hi, >> >> I am trying to setup 82576 VF passthrough to pv domU non >Vt-d machineYou can''t run SR-IOV VF devices in VM guests on a non Vt-d machine. VT-d is required for VF passthrough to a guest. - Greg LAD Intel Corp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I remember the VF driver need hook for configuration space access. For example, the BAR is 0 by default. Not sure if that's the reason. --jyh ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Rose, Gregory V Sent: 2009年9月25日 13:46 To: Satish Chowdhury Cc: Simon Horman; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] 82576 VF passthrough I've only ever used HVMs. I can't say whether VF passthrough to a PV guest works or not but I can't think of any reason it wouldn't. Can you pass other devices through to this PV guest? If so then a VF should work also. Generally any guest that you can a physical device through will also work with a VF device. Next thing to consider is whether your BIOS supports SR-IOV. There are a lot of machines that have the necessary HW to support SR-IOV but the BIOS does not properly detect SR-IOV capable devices and program the upstream switches and bridges properly to make sure the I/O aperture is large enough to accommodate the extra VF devices when they are enabled by the driver. There are very few BIOS vendors shipping SR-IOV capable BIOS at the moment. If your machine does not have the right BIOS then there is a work around in the Xen 3.4 release that you can use. Or at least I think it's still there. pci=assign-mmio=0000:01 Try adding that to your kernel command line but substitute the domain/bus of your device. For instance if your Kawela is on bus 4 then you would change the command to pci=assign-mmio=0000:04. Hope that helps. - Greg ________________________________ From: Satish Chowdhury [mailto:satish.chowdhury@oneconvergence.com] Sent: Thursday, September 24, 2009 10:32 PM To: Rose, Gregory V Cc: Simon Horman; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] 82576 VF passthrough Hi Greg, I am using Vt-d machine. Sorry for the confusion. My Hardware is HP dc7800p Chipset: Q35, with Intel Core 2 Duo E8400 processor. NIC Card is 82576 E1G42ET (dual port RJ-45 copper). Is VF passthrough can be done only to HVM guest? Is VF passthrough to paravirtal guest is supported? Regards, -Satish On Fri, Sep 25, 2009 at 10:36 AM, Rose, Gregory V <gregory.v.rose@intel.com<mailto:gregory.v.rose@intel.com>> wrote:>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com> >[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>] On Behalf Of >Simon Horman >Sent: Wednesday, September 23, 2009 4:25 PM >To: Satish Chowdhury >Cc: xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com> >Subject: Re: [Xen-devel] 82576 VF passthrough > >On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote: >> Hi, >> >> I am trying to setup 82576 VF passthrough to pv domU non >Vt-d machineYou can't run SR-IOV VF devices in VM guests on a non Vt-d machine. VT-d is required for VF passthrough to a guest. - Greg LAD Intel Corp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Sep 24, 2009 at 10:46:12PM -0700, Rose, Gregory V wrote:> I''ve only ever used HVMs. I can''t say whether VF passthrough to a PV guest works or not but I can''t think of any reason it wouldn''t. Can you pass other devices through to this PV guest? If so then a VF should work also. Generally any guest that you can a physical device through will also work with a VF device. > > Next thing to consider is whether your BIOS supports SR-IOV. There are a lot of machines that have the necessary HW to support SR-IOV but the BIOS does not properly detect SR-IOV capable devices and program the upstream switches and bridges properly to make sure the I/O aperture is large enough to accommodate the extra VF devices when they are enabled by the driver. There are very few BIOS vendors shipping SR-IOV capable BIOS at the moment. If your machine does not have the right BIOS then there is a work around in the Xen 3.4 release that you can use. Or at least I think it''s still there. > > pci=assign-mmio=0000:01 > > Try adding that to your kernel command line but substitute the domain/bus of your device. For instance if your Kawela is on bus 4 then you would change the command to pci=assign-mmio=0000:04.Hi, have tried pass-through with very similar hardware to Sanish, though only with HVM domains. The bios on my dc7800 does not seem to set up VT-d correctly and I found that the following boot parameter was required on order to allow the igb driver to create VFs. pci=assign-busses _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel