I have successfully booted xen on Fedora 14 x86_64 with iommu=force on the kernel line, I can unbind a pci device and rebind to pci-stub and see it with xm pci-list-assignable-devices. That''s cool. Now I have an Intel 82576 dual-port ET Adapter that supports SR-IOV and started configuring everything to enable it''s virtual functions: added pci_pt_e820_access=on to the module line that contains the vmlinuz created igb.conf under /etc/modprobe.d with options igb max_vfs=7 added blacklist igbvf to /etc/modprobe.d/blacklist.conf I was under the impression that I should see some kind of message about MMCONFIG in xm dmesg, but there isn''t anything. I do a xm dmesg | grep -i pci and it finds nothing with MMCONFIG. I guess my next question is what do I do next? How do I troubleshoot further? Below is the VT-d messages from xm dmesg: (XEN) Intel VT-d Snoop Control supported. (XEN) Intel VT-d DMA Passthrough not supported. (XEN) Intel VT-d Queued Invalidation supported. (XEN) Intel VT-d Interrupt Remapping not supported. Does MMCONFIG have something to do with the fact the Interrupt Remapping not supported message is displayed? Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Dec-08 16:32 UTC
Re: [Xen-devel] Intel S3420GPLX VT-d VF question
On Tue, Dec 07, 2010 at 11:52:24PM -0500, Michael A. Collins wrote:> I have successfully booted xen on Fedora 14 x86_64 with iommu=forceOk.. Um, what Fedora kernel are we taking about here? Do you have the full serial log?> on the kernel line, I can unbind a pci device and rebind to pci-stub > and see it with xm pci-list-assignable-devices. That''s cool. Now I > have an Intel 82576 dual-port ET Adapter that supports SR-IOV and > started configuring everything to enable it''s virtual functions: > added pci_pt_e820_access=on to the module line that contains the > vmlinuzWhy? It doesn''t automatically load without that parameter?> created igb.conf under /etc/modprobe.d with options igb max_vfs=7 > added blacklist igbvf to /etc/modprobe.d/blacklist.confWhy blacklist it?> > I was under the impression that I should see some kind of message > about MMCONFIG in xm dmesg, but there isn''t anything. I do a xm > dmesg | grep -i pci and it finds nothing with MMCONFIG. I guess myHuh? Why? Do you see the 7 ethernet PCI device being created?> next question is what do I do next? How do I troubleshoot further?What are you trying to troubleshoot? Did you lspci output change after you loaded the igbvf?> Below is the VT-d messages from xm dmesg: > (XEN) Intel VT-d Snoop Control supported. > (XEN) Intel VT-d DMA Passthrough not supported. > (XEN) Intel VT-d Queued Invalidation supported. > (XEN) Intel VT-d Interrupt Remapping not supported. > > Does MMCONFIG have something to do with the fact the Interrupt > Remapping not supported message is displayed?No. It just means your VT-D chipset does not support interrupt remapping.> Mike > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 8 Dec 2010 11:32:18 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Tue, Dec 07, 2010 at 11:52:24PM -0500, Michael A. Collins wrote: >> I have successfully booted xen on Fedora 14 x86_64 with iommu=force > > Ok.. Um, what Fedora kernel are we taking about here? Do you have > the full serial log? >> on the kernel line, I can unbind a pci device and rebind to pci-stub >> and see it with xm pci-list-assignable-devices. That''s cool. Now I >> have an Intel 82576 dual-port ET Adapter that supports SR-IOV and >> started configuring everything to enable it''s virtual functions: >> added pci_pt_e820_access=on to the module line that contains the >> vmlinuz > > Why? It doesn''t automatically load without that parameter? >> created igb.conf under /etc/modprobe.d with options igb max_vfs=7 >> added blacklist igbvf to /etc/modprobe.d/blacklist.conf > > Why blacklist it? >> >> I was under the impression that I should see some kind of message >> about MMCONFIG in xm dmesg, but there isn''t anything. I do a xm >> dmesg | grep -i pci and it finds nothing with MMCONFIG. I guess my > > Huh? Why? Do you see the 7 ethernet PCI device being created? > >> next question is what do I do next? How do I troubleshoot further? > > What are you trying to troubleshoot? Did you lspci output change > after you loaded the igbvf? > >> Below is the VT-d messages from xm dmesg: >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping not supported. >> >> Does MMCONFIG have something to do with the fact the Interrupt >> Remapping not supported message is displayed? > > No. It just means your VT-D chipset does not support interrupt > remapping. > >> Mike >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-develI didn''t know if this actually made it on the list, since it didn''t show up in my inbox when I sent it, so the second email from me is a duplicate. Sorry. The kernel I am using is myoung''s pre-built rpm from http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm. I am also using the 4.0.1 rpms from http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm. I am following the instructions from http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial. The pci_pt_e820_access=on setting was taken from the following blog: http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html. I was under the impression that that option had to be set for the VFs to be seen by the kernel. I have not tried it without, but will do that tonight. The blacklist settings was referenced in the above article, so I stuck it in. I will try to load the igbvf tonight as well. I do not see the 7 Virtual Functions in lspci. Thanks for the reply, I will try without the e820 option and loading the igbvf module tonight if that works, I''ll let you know. If not, I''ll respond with the full xen dmesg output. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Dec-08 17:50 UTC
Re: [Xen-devel] Intel S3420GPLX VT-d VF question
> The kernel I am using is myoung''s pre-built rpm from http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm.Sweet. That is quite up-to-date.> I am also using the 4.0.1 rpms from > http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.Uhh, try a more modern version. The xen-unstable.hg - you will have to compile it yourself. Pasi''s awesome PVOPS Wiki mentions the appropiate steps and I think Boris''s blog has details as well.> I am following the instructions from > http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial. > > The pci_pt_e820_access=on setting was taken from the following blog: http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.Oh gosh. That is with 2.6.18 and Xen 3.4.> I was under the impression that that option had to be set for the > VFs to be seen by the kernel.Well, they were workarounds for that verison of kernel. It might be that they are required in 2.6.32.. but lets assume that those are actually fixed.> I have not tried it without, but will do that tonight. The > blacklist settings was referenced in the above article, so I stuck > it in. I will try to load the igbvf tonight as well.Yeah, try it. If it works you should see suddenly seven extra NICs (ifconfig -a) in your box. If that works then we know the kernel driver is working.> > I do not see the 7 Virtual Functions in lspci.Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)" must show up. That is the proof that your max_ig.. parameter worked and the card is now able to do SR-IOV. You might have need to upgrade your lspci - I think I had it as "Unknown device" at some point. Anyhow, if you see those extra devices, _then_ you can blacklist the igvf module.> > Thanks for the reply, I will try without the e820 option and loading > the igbvf module tonight if that works, I''ll let you know. If not, > I''ll respond with the full xen dmesg output.Take your time. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> The kernel I am using is myoung''s pre-built rpm from >> http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm. > > Sweet. That is quite up-to-date. > >> I am also using the 4.0.1 rpms from >> http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm. > > Uhh, try a more modern version. The xen-unstable.hg - you will have > to compile it yourself. Pasi''s awesome PVOPS Wiki mentions the > appropiate > steps and I think Boris''s blog has details as well. > >> I am following the instructions from >> http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial. >> >> The pci_pt_e820_access=on setting was taken from the following blog: >> http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html. > > Oh gosh. That is with 2.6.18 and Xen 3.4. > >> I was under the impression that that option had to be set for the >> VFs to be seen by the kernel. > > Well, they were workarounds for that verison of kernel. It might be > that they are required in 2.6.32.. but lets assume that those > are actually fixed. > >> I have not tried it without, but will do that tonight. The >> blacklist settings was referenced in the above article, so I stuck >> it in. I will try to load the igbvf tonight as well. > > Yeah, try it. If it works you should see suddenly seven extra NICs > (ifconfig -a) in your box. If that works then we know the kernel > driver is working. >> >> I do not see the 7 Virtual Functions in lspci. > > Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576 > Gigabit Network Connection (rev 01) > 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01)" > > must show up. That is the proof that your max_ig.. parameter worked > and the card > is now able to do SR-IOV. You might have need to upgrade your lspci - > I think I had > it as "Unknown device" at some point. > > Anyhow, if you see those extra devices, _then_ you can blacklist the > igvf module. > >> >> Thanks for the reply, I will try without the e820 option and loading >> the igbvf module tonight if that works, I''ll let you know. If not, >> I''ll respond with the full xen dmesg output. > > Take your time.Ok, I did try to remove the e820 option and got rid of the igbvf blacklisting. No change, the Virtual Function entries are not showing up. I beleieve that I may be jumping ahead of myself. I am going to blow away the box and reload with Fedora 14 again, but not move forward with the Xen install until after I verify that SR-IOV works as it should. I will keep you posted, but I probably won''t start till this weekend. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> The kernel I am using is myoung''s pre-built rpm from >> http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm. > > Sweet. That is quite up-to-date. > >> I am also using the 4.0.1 rpms from >> http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm. > > Uhh, try a more modern version. The xen-unstable.hg - you will have > to compile it yourself. Pasi''s awesome PVOPS Wiki mentions the > appropiate > steps and I think Boris''s blog has details as well. > >> I am following the instructions from >> http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial. >> >> The pci_pt_e820_access=on setting was taken from the following blog: >> http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html. > > Oh gosh. That is with 2.6.18 and Xen 3.4. > >> I was under the impression that that option had to be set for the >> VFs to be seen by the kernel. > > Well, they were workarounds for that verison of kernel. It might be > that they are required in 2.6.32.. but lets assume that those > are actually fixed. > >> I have not tried it without, but will do that tonight. The >> blacklist settings was referenced in the above article, so I stuck >> it in. I will try to load the igbvf tonight as well. > > Yeah, try it. If it works you should see suddenly seven extra NICs > (ifconfig -a) in your box. If that works then we know the kernel > driver is working. >> >> I do not see the 7 Virtual Functions in lspci. > > Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576 > Gigabit Network Connection (rev 01) > 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01) > 01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function > (rev 01)" > > must show up. That is the proof that your max_ig.. parameter worked > and the card > is now able to do SR-IOV. You might have need to upgrade your lspci - > I think I had > it as "Unknown device" at some point. > > Anyhow, if you see those extra devices, _then_ you can blacklist the > igvf module. > >> >> Thanks for the reply, I will try without the e820 option and loading >> the igbvf module tonight if that works, I''ll let you know. If not, >> I''ll respond with the full xen dmesg output. > > Take your time.Seems as the igb module was loaded in the initramfs and was not reloaded using the options specified in my igb.conf file under /etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and suddenly all the Virtual Functions showed up. So that was the problem all along. Until I figure out how to edit the initramfs to load igb with max_vfs option, I''m putting the rmmod and modprobe in the rc.local file, that seems to work well and it gets executed before xend. I am build xen-4.0-testing to replace the rpms I used. Once all that is done and I test passing through one of the Virtual Functions to a vm, I''ll report back. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> >> Thanks for the reply, I will try without the e820 option and loading >> the igbvf module tonight if that works, I''ll let you know. If not, >> I''ll respond with the full xen dmesg output. > > Take your time.Ok, got a test vm running and the igbvf module loaded, so it looks like everything is working. I can''t seem to get any traffic sent or received from it, but I''ll work through that eventually. Thanks for all your help! Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" <mike@ark-net.org> wrote:> On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> Take your time. > > Seems as the igb module was loaded in the initramfs and was not > reloaded using the options specified in my igb.conf file under > /etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and > suddenly all the Virtual Functions showed up. So that was the > problem > all along. Until I figure out how to edit the initramfs to load igb > with max_vfs option, I''m putting the rmmod and modprobe in the > rc.local file, that seems to work well and it gets executed before > xend. I am build xen-4.0-testing to replace the rpms I used. Once > all that is done and I test passing through one of the Virtual > Functions to a vm, I''ll report back. > MikeOk, so I can pass one of the virtual functions through to a HVM. I think I''m still not out of the water though. It''s acting very strangely and I have not successfully been able to very layer-2 connectivity with it. I am seeing the following in my HVM''s qemu log: register_real_device: Assigning real physical device 04:10.4 ... pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function pt_iomul_init: Error: pt_iomul_init can''t open file /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4 pt_register_regions: IO region registered (size=0x00004000 base_addr=0xb2448004) pt_register_regions: IO region registered (size=0x00004000 base_addr=0xb2468004) pt_msix_init: get MSI-X table bar base b2468000 pt_msix_init: table_off = 0, total_entries = 3 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7fe3c4359000 register_real_device: Real physical device 04:10.4 registered successfuly! IRQ type = INTx generate a sci for PHP. deassert due to disable GPE bit. pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:30h][Length:4] pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 index=0 first_map=1 pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 index=3 first_map=1 Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which leads me to believe that the pass through isn''t actually working. I am still using the mayoung built xen-4.0.1-6 rpms from koji, since I have not been able to successfully compile xen stubdom. I guess gcc 4.5 is just too new and untested for me to troubleshoot and get through. Hopefully, someone on here can point me in the right direction. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Dec-22 20:10 UTC
Re: [Xen-devel] Intel S3420GPLX VT-d VF question
On Mon, Dec 20, 2010 at 09:24:17PM -0500, Michael A. Collins wrote:> On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" > <mike@ark-net.org> wrote: > >On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk > ><konrad.wilk@oracle.com> wrote: > >>Take your time. > > > >Seems as the igb module was loaded in the initramfs and was not > >reloaded using the options specified in my igb.conf file under > >/etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and > >suddenly all the Virtual Functions showed up. So that was the > >problem > >all along. Until I figure out how to edit the initramfs to load igb > >with max_vfs option, I''m putting the rmmod and modprobe in the > >rc.local file, that seems to work well and it gets executed before > >xend. I am build xen-4.0-testing to replace the rpms I used. Once > >all that is done and I test passing through one of the Virtual > >Functions to a vm, I''ll report back. > >Mike > > > Ok, so I can pass one of the virtual functions through to a HVM. I > think I''m still not out of the water though. It''s acting very > strangely and I have not successfully been able to very layer-2And if you tried to use the device from dom0, or even baremetal, you had not trouble with the VIFs, right?> connectivity with it. I am seeing the following in my HVM''s qemu > log: > register_real_device: Assigning real physical device 04:10.4 ... > pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function > pt_iomul_init: Error: pt_iomul_init can''t open file > /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4 > pt_register_regions: IO region registered (size=0x00004000 > base_addr=0xb2448004) > pt_register_regions: IO region registered (size=0x00004000 > base_addr=0xb2468004) > pt_msix_init: get MSI-X table bar base b2468000 > pt_msix_init: table_off = 0, total_entries = 3 > pt_msix_init: errno = 2^^^^^^^ That does not sound good, but it might be just b/c it could not find /dev/xen/pci_iomul.> pt_msix_init: mapping physical MSI-X table to 7fe3c4359000 > register_real_device: Real physical device 04:10.4 registered > successfuly! > IRQ type = INTx > generate a sci for PHP. > deassert due to disable GPE bit. > pt_pci_write_config: Warning: Guest attempt to set address to unused > Base Address Register. [00:06.0][Offset:30h][Length:4] > pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 > index=0 first_map=1 > pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 > index=3 first_map=1 > > > Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which > leads me to believe that the pass through isn''t actually working. I > am still using the mayoung built xen-4.0.1-6 rpms from koji, since I > have not been able to successfully compile xen stubdom. I guess gcc > 4.5 is just too new and untested for me to troubleshoot and get > through. Hopefully, someone on here can point me in the right > direction.Hmm. I am bit out of ideas.. Do you know which line in the QEMU code would trigger that? You could instrument QEMU a bit? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 22 Dec 2010 15:10:12 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Mon, Dec 20, 2010 at 09:24:17PM -0500, Michael A. Collins wrote: >> On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" >> <mike@ark-net.org> wrote: >> >On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk >> ><konrad.wilk@oracle.com> wrote: >> >>Take your time. >> > >> >Seems as the igb module was loaded in the initramfs and was not >> >reloaded using the options specified in my igb.conf file under >> >/etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and >> >suddenly all the Virtual Functions showed up. So that was the >> >problem >> >all along. Until I figure out how to edit the initramfs to load >> igb >> >with max_vfs option, I''m putting the rmmod and modprobe in the >> >rc.local file, that seems to work well and it gets executed before >> >xend. I am build xen-4.0-testing to replace the rpms I used. Once >> >all that is done and I test passing through one of the Virtual >> >Functions to a vm, I''ll report back. >> >Mike >> >> >> Ok, so I can pass one of the virtual functions through to a HVM. I >> think I''m still not out of the water though. It''s acting very >> strangely and I have not successfully been able to very layer-2 > > And if you tried to use the device from dom0, or even baremetal, you > had not trouble with the VIFs, right? > >> connectivity with it. I am seeing the following in my HVM''s qemu >> log: >> register_real_device: Assigning real physical device 04:10.4 ... >> pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function >> pt_iomul_init: Error: pt_iomul_init can''t open file >> /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4 >> pt_register_regions: IO region registered (size=0x00004000 >> base_addr=0xb2448004) >> pt_register_regions: IO region registered (size=0x00004000 >> base_addr=0xb2468004) >> pt_msix_init: get MSI-X table bar base b2468000 >> pt_msix_init: table_off = 0, total_entries = 3 >> pt_msix_init: errno = 2 > ^^^^^^^ > That does not sound good, but it might be just b/c it could > not find /dev/xen/pci_iomul. > >> pt_msix_init: mapping physical MSI-X table to 7fe3c4359000 >> register_real_device: Real physical device 04:10.4 registered >> successfuly! >> IRQ type = INTx >> generate a sci for PHP. >> deassert due to disable GPE bit. >> pt_pci_write_config: Warning: Guest attempt to set address to unused >> Base Address Register. [00:06.0][Offset:30h][Length:4] >> pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 >> index=0 first_map=1 >> pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 >> index=3 first_map=1 >> >> >> Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which >> leads me to believe that the pass through isn''t actually working. I >> am still using the mayoung built xen-4.0.1-6 rpms from koji, since I >> have not been able to successfully compile xen stubdom. I guess gcc >> 4.5 is just too new and untested for me to troubleshoot and get >> through. Hopefully, someone on here can point me in the right >> direction. > > Hmm. I am bit out of ideas.. Do you know which line in the QEMU code > would trigger that? You could instrument QEMU a bit?Yeah, the VIFs are fine in dom0. Also, I have seen reports of other people receiving the /dev/xen/pci_iomul and still getting past it. I will dig around in the QEMU code for awhile and see what I can come up with. Thanks. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Can you attach some logs by dmesg in your guest just after *igbvf* module inserted? Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you ever checked the physical ports in your card? If you the card takes two ports, generally, one is for odd and the other is for even function of the BDF. Best Regards Zhuo Song Gif me a visky, ginger ale on the side. --------- On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" <mike@ark-net.org> wrote:> On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> Take your time. > > Seems as the igb module was loaded in the initramfs and was not > reloaded using the options specified in my igb.conf file under > /etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and > suddenly all the Virtual Functions showed up. So that was the > problem > all along. Until I figure out how to edit the initramfs to load igb > with max_vfs option, I''m putting the rmmod and modprobe in the > rc.local file, that seems to work well and it gets executed before > xend. I am build xen-4.0-testing to replace the rpms I used. Once > all that is done and I test passing through one of the Virtual > Functions to a vm, I''ll report back. > MikeOk, so I can pass one of the virtual functions through to a HVM. I think I''m still not out of the water though. It''s acting very strangely and I have not successfully been able to very layer-2 connectivity with it. I am seeing the following in my HVM''s qemu log: register_real_device: Assigning real physical device 04:10.4 ... pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function pt_iomul_init: Error: pt_iomul_init can''t open file /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4 pt_register_regions: IO region registered (size=0x00004000 base_addr=0xb2448004) pt_register_regions: IO region registered (size=0x00004000 base_addr=0xb2468004) pt_msix_init: get MSI-X table bar base b2468000 pt_msix_init: table_off = 0, total_entries = 3 pt_msix_init: errno = 2 pt_msix_init: mapping physical MSI-X table to 7fe3c4359000 register_real_device: Real physical device 04:10.4 registered successfuly! IRQ type = INTx generate a sci for PHP. deassert due to disable GPE bit. pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:30h][Length:4] pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 index=0 first_map=1 pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 index=3 first_map=1 Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which leads me to believe that the pass through isn''t actually working. I am still using the mayoung built xen-4.0.1-6 rpms from koji, since I have not been able to successfully compile xen stubdom. I guess gcc 4.5 is just too new and untested for me to troubleshoot and get through. Hopefully, someone on here can point me in the right direction. Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 24 Dec 2010 13:38:36 +0800, "Zhuo Song" <alfred.z.song@gmail.com> wrote:> Can you attach some logs by dmesg in your guest just after *igbvf* > module inserted? > Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you > ever checked the physical ports in your card? > If you the card takes two ports, generally, one is for odd and the > other is for even function of the BDF. > > Best Regards > > Zhuo Song > Gif me a visky, ginger ale on the side. >Dec 26 19:42:35 localhost kernel: [ 450.895517] pci 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment Dec 26 19:42:35 localhost kernel: [ 450.895528] pci 0000:00:07.0: BAR 0: assigned [mem 0x40000000-0x40003fff 64bit] Dec 26 19:42:35 localhost kernel: [ 450.895986] pci 0000:00:07.0: BAR 0: set to [mem 0x40000000-0x40003fff 64bit] (PCI address [0x40000000-0x40003fff] Dec 26 19:42:35 localhost kernel: [ 450.895999] pci 0000:00:07.0: BAR 3: assigned [mem 0x40004000-0x40007fff 64bit] Dec 26 19:42:35 localhost kernel: [ 450.896513] pci 0000:00:07.0: BAR 3: set to [mem 0x40004000-0x40007fff 64bit] (PCI address [0x40004000-0x40007fff] Dec 26 19:42:35 localhost kernel: [ 450.902103] pci 0000:00:00.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902108] pci 0000:00:00.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.902446] pci 0000:00:01.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902452] pci 0000:00:01.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.902869] ata_piix 0000:00:01.1: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902875] ata_piix 0000:00:01.1: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.903527] uhci_hcd 0000:00:01.2: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.903532] uhci_hcd 0000:00:01.2: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.903942] pci 0000:00:01.3: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.903948] pci 0000:00:01.3: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.904549] pci 0000:00:02.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.904554] pci 0000:00:02.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.904930] pci 0000:00:03.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.904933] pci 0000:00:03.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.905213] 8139cp 0000:00:04.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.905215] 8139cp 0000:00:04.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.905366] pci 0000:00:07.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 451.038332] Intel(R) Virtual Function Network Driver - version 1.0.0-k0 Dec 26 19:42:35 localhost kernel: [ 451.038334] Copyright (c) 2009 Intel Corporation. Dec 26 19:42:35 localhost kernel: [ 451.038413] igbvf 0000:00:07.0: enabling device (0000 -> 0002) Dec 26 19:42:35 localhost kernel: [ 451.070856] igbvf 0000:00:07.0: Intel(R) 82576 Virtual Function Dec 26 19:42:35 localhost kernel: [ 451.070865] igbvf 0000:00:07.0: Address: 82:bb:48:c2:5f:ca Dec 26 19:42:35 localhost kernel: [ 451.070871] igbvf 0000:00:07.0: MAC: 1 Dec 26 19:42:42 localhost kernel: [ 457.800481] pci 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment Dec 26 19:42:42 localhost kernel: [ 457.800492] pci 0000:00:06.0: BAR 0: assigned [mem 0x40008000-0x4000bfff 64bit] Dec 26 19:42:42 localhost kernel: [ 457.800922] pci 0000:00:06.0: BAR 0: set to [mem 0x40008000-0x4000bfff 64bit] (PCI address [0x40008000-0x4000bfff] Dec 26 19:42:42 localhost kernel: [ 457.800929] pci 0000:00:06.0: BAR 3: assigned [mem 0x4000c000-0x4000ffff 64bit] Dec 26 19:42:42 localhost kernel: [ 457.801416] pci 0000:00:06.0: BAR 3: set to [mem 0x4000c000-0x4000ffff 64bit] (PCI address [0x4000c000-0x4000ffff] Dec 26 19:42:42 localhost kernel: [ 457.801744] pci 0000:00:00.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.801748] pci 0000:00:00.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804131] pci 0000:00:01.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804136] pci 0000:00:01.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804405] ata_piix 0000:00:01.1: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804409] ata_piix 0000:00:01.1: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804662] uhci_hcd 0000:00:01.2: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804665] uhci_hcd 0000:00:01.2: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804917] pci 0000:00:01.3: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804920] pci 0000:00:01.3: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809453] pci 0000:00:02.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809455] pci 0000:00:02.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809646] pci 0000:00:03.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809648] pci 0000:00:03.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809758] 8139cp 0000:00:04.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809759] 8139cp 0000:00:04.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809888] igbvf 0000:00:07.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809892] pci 0000:00:06.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.864627] igbvf 0000:00:06.0: enabling device (0000 -> 0002) Dec 26 19:42:42 localhost kernel: [ 457.898412] igbvf 0000:00:06.0: Intel(R) 82576 Virtual Function Dec 26 19:42:42 localhost kernel: [ 457.898416] igbvf 0000:00:06.0: Address: ca:c0:62:de:59:c7 Dec 26 19:42:42 localhost kernel: [ 457.898419] igbvf 0000:00:06.0: MAC: 1 Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Not so sure. But I think you first need to check into your guest OS for that the ethernet port which bound to your VF is UP.( after it''s been UP, you can see dmesg for some info, like some words "eth0/1 is ready") The related PF in the dom0 is UP too. It''s mentioned that, like the previous talk, If the card takes two ports, generally, one is for odd and the other is for even function of the VF''s BDF(lspci in dom0) which you passed thru. So one way is to try both ports. Best Regards Alfred Song Gif me a visky, ginger ale on the side. On Fri, 24 Dec 2010 13:38:36 +0800, "Zhuo Song" <alfred.z.song@gmail.com> wrote:> Can you attach some logs by dmesg in your guest just after *igbvf* > module inserted? > Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you > ever checked the physical ports in your card? > If you the card takes two ports, generally, one is for odd and the > other is for even function of the BDF. > > Best Regards > > Zhuo Song > Gif me a visky, ginger ale on the side. >Dec 26 19:42:35 localhost kernel: [ 450.895517] pci 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment Dec 26 19:42:35 localhost kernel: [ 450.895528] pci 0000:00:07.0: BAR 0: assigned [mem 0x40000000-0x40003fff 64bit] Dec 26 19:42:35 localhost kernel: [ 450.895986] pci 0000:00:07.0: BAR 0: set to [mem 0x40000000-0x40003fff 64bit] (PCI address [0x40000000-0x40003fff] Dec 26 19:42:35 localhost kernel: [ 450.895999] pci 0000:00:07.0: BAR 3: assigned [mem 0x40004000-0x40007fff 64bit] Dec 26 19:42:35 localhost kernel: [ 450.896513] pci 0000:00:07.0: BAR 3: set to [mem 0x40004000-0x40007fff 64bit] (PCI address [0x40004000-0x40007fff] Dec 26 19:42:35 localhost kernel: [ 450.902103] pci 0000:00:00.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902108] pci 0000:00:00.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.902446] pci 0000:00:01.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902452] pci 0000:00:01.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.902869] ata_piix 0000:00:01.1: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.902875] ata_piix 0000:00:01.1: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.903527] uhci_hcd 0000:00:01.2: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.903532] uhci_hcd 0000:00:01.2: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.903942] pci 0000:00:01.3: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.903948] pci 0000:00:01.3: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.904549] pci 0000:00:02.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.904554] pci 0000:00:02.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.904930] pci 0000:00:03.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.904933] pci 0000:00:03.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.905213] 8139cp 0000:00:04.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 450.905215] 8139cp 0000:00:04.0: using default PCI settings Dec 26 19:42:35 localhost kernel: [ 450.905366] pci 0000:00:07.0: no hotplug settings from platform Dec 26 19:42:35 localhost kernel: [ 451.038332] Intel(R) Virtual Function Network Driver - version 1.0.0-k0 Dec 26 19:42:35 localhost kernel: [ 451.038334] Copyright (c) 2009 Intel Corporation. Dec 26 19:42:35 localhost kernel: [ 451.038413] igbvf 0000:00:07.0: enabling device (0000 -> 0002) Dec 26 19:42:35 localhost kernel: [ 451.070856] igbvf 0000:00:07.0: Intel(R) 82576 Virtual Function Dec 26 19:42:35 localhost kernel: [ 451.070865] igbvf 0000:00:07.0: Address: 82:bb:48:c2:5f:ca Dec 26 19:42:35 localhost kernel: [ 451.070871] igbvf 0000:00:07.0: MAC: 1 Dec 26 19:42:42 localhost kernel: [ 457.800481] pci 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment Dec 26 19:42:42 localhost kernel: [ 457.800492] pci 0000:00:06.0: BAR 0: assigned [mem 0x40008000-0x4000bfff 64bit] Dec 26 19:42:42 localhost kernel: [ 457.800922] pci 0000:00:06.0: BAR 0: set to [mem 0x40008000-0x4000bfff 64bit] (PCI address [0x40008000-0x4000bfff] Dec 26 19:42:42 localhost kernel: [ 457.800929] pci 0000:00:06.0: BAR 3: assigned [mem 0x4000c000-0x4000ffff 64bit] Dec 26 19:42:42 localhost kernel: [ 457.801416] pci 0000:00:06.0: BAR 3: set to [mem 0x4000c000-0x4000ffff 64bit] (PCI address [0x4000c000-0x4000ffff] Dec 26 19:42:42 localhost kernel: [ 457.801744] pci 0000:00:00.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.801748] pci 0000:00:00.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804131] pci 0000:00:01.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804136] pci 0000:00:01.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804405] ata_piix 0000:00:01.1: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804409] ata_piix 0000:00:01.1: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804662] uhci_hcd 0000:00:01.2: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804665] uhci_hcd 0000:00:01.2: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.804917] pci 0000:00:01.3: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.804920] pci 0000:00:01.3: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809453] pci 0000:00:02.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809455] pci 0000:00:02.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809646] pci 0000:00:03.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809648] pci 0000:00:03.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809758] 8139cp 0000:00:04.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809759] 8139cp 0000:00:04.0: using default PCI settings Dec 26 19:42:42 localhost kernel: [ 457.809888] igbvf 0000:00:07.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.809892] pci 0000:00:06.0: no hotplug settings from platform Dec 26 19:42:42 localhost kernel: [ 457.864627] igbvf 0000:00:06.0: enabling device (0000 -> 0002) Dec 26 19:42:42 localhost kernel: [ 457.898412] igbvf 0000:00:06.0: Intel(R) 82576 Virtual Function Dec 26 19:42:42 localhost kernel: [ 457.898416] igbvf 0000:00:06.0: Address: ca:c0:62:de:59:c7 Dec 26 19:42:42 localhost kernel: [ 457.898419] igbvf 0000:00:06.0: MAC: 1 Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel