Hi group members, I am kind of confused with SRIOV. It will be very great if any of you put some light on it. Does SRIOV and PCIe passthrough are same when it comes to direct assignment? If we have both PCI and PCIe NIC card in our system how can we do PCI passthrough in HVM domu? I know about PCI passthrough and want to know about PCIe and SRIOV any tutorial or documentation on it? I tried to do SRIOV on Xen 4.0 with Ubuntu 10.04and my HVM is also Ubuntu 10.04 when I do lspci it does not show virtual function in it. Any help will be appreciated. Thank you. Annie _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Annie,
Been doing some reading up on SR-IOV as well. Maybe I can help.
AFAIK, SR-IOV depends on VT-d and works via PCIe passthrough. In
other words, the PCI device exposes virtual functions which you can
assign to a VM.
For Intel network adapters, you have to load the igb driver with
the max_vfs=[number] option for the virtual functions to appear. Here
are two step-by-step guides you can refer to:
http://www.intel.com/content/dam/doc/technology-brief/ethernet-SR-IOV-tech-brief.pdf
http://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe
The links use RHEL as the dom0, but it should be easy to adapt
them to Ubuntu.
Regarding your question about passing through PCI and PCIe cards
to a HVM; one, you will need VT-d or AMD-Vi for passthrough to HVM to
work; two, if I''m not wrong, PCI passthrough does not differentiate
between PCI and PCIe devices -- except special cases when conflicts
arise.
Hope this helps!
Hello, On 24 November 2011 02:36, Achala Aryal <arya2595@vandals.uidaho.edu> wrote:> thank you for your respond. > I am still not seeing any Virtual function when I do lspci. > > 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B > PCI Express Gigabit Ethernet controller (rev 03) > Subsystem: ASRock Incorporation Device 8168 > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- > Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 32 bytes > Interrupt: pin A routed to IRQ 2245 > Region 0: I/O ports at c800 [size=256] > Region 2: Memory at cfdff000 (64-bit, prefetchable) [size=4K] > Region 4: Memory at cfdf8000 (64-bit, prefetchable) [size=16K] > Expansion ROM at f7ee0000 [disabled] [size=128K] > Capabilities: [40] Power Management version 3 > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA > PME(D0+,D1+,D2+,D3hot+,D3cold+) > Status: D0 PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ > Queue=0/0 Enable+ > Address: 00000000feeff00c Data: 4151 > Capabilities: [70] Express (v2) Endpoint, MSI 01 > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s > <512ns, L1 <64us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > Unsupported- > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 128 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ > TransPend- > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, > Latency L0 <512ns, L1 <64us > ClockPM+ Suprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ > DLActive- BWMgmt- ABWMgmt- > Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4 > Vector table: BAR=4 offset=00000000 > PBA: BAR=4 offset=00000800 > Capabilities: [cc] Vital Product Data <?> > Capabilities: [100] Advanced Error Reporting <?> > Capabilities: [140] Virtual Channel <?> > Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-00-03 > Kernel driver in use: r8168 > Kernel modules: r8168, r8169 > > > do you think I am missing something??? > > Your help will be appreciated. > Thank you > >Please remember to CC the xen-users list! According to the NIC''s page: http://www.realtek.com/products/productsView.aspx?Langid=1&PNid=13&PFid=5&Level=5&Conn=4&ProdID=12 Your controller does not have SR-IOV! AFAIK, only some Intel controllers have SR-IOV: (Pause at 9:09, SR-IOV is on the right side of that list, not the left side) http://www.youtube.com/watch?v=hRHsk8Nycdg#t=9m9s You can check which controller has SR-IOV by checking its page at http://ark.intel.com Hope this helps
Oh got it. So If I want to passthrough this device (which is PCIe not PCI ) then is it a same procedure like in PCI passthrough hiding it from dom0 and assigning it to domU config flie.... also one more since this is pcie device, I can passthrough to more than one domU right? Thank you> From: xieliwei@gmail.com > Date: Thu, 24 Nov 2011 02:51:03 +0800 > To: arya2595@vandals.uidaho.edu > CC: xen-users@lists.xensource.com > Subject: Re: [Xen-users] SR-IOV > > Hello, > > On 24 November 2011 02:36, Achala Aryal <arya2595@vandals.uidaho.edu> wrote: > > thank you for your respond. > > I am still not seeing any Virtual function when I do lspci. > > > > 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B > > PCI Express Gigabit Ethernet controller (rev 03) > > Subsystem: ASRock Incorporation Device 8168 > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- > > Stepping- SERR- FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > > <TAbort- <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 32 bytes > > Interrupt: pin A routed to IRQ 2245 > > Region 0: I/O ports at c800 [size=256] > > Region 2: Memory at cfdff000 (64-bit, prefetchable) [size=4K] > > Region 4: Memory at cfdf8000 (64-bit, prefetchable) [size=16K] > > Expansion ROM at f7ee0000 [disabled] [size=128K] > > Capabilities: [40] Power Management version 3 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA > > PME(D0+,D1+,D2+,D3hot+,D3cold+) > > Status: D0 PME-Enable- DSel=0 DScale=0 PME- > > Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ > > Queue=0/0 Enable+ > > Address: 00000000feeff00c Data: 4151 > > Capabilities: [70] Express (v2) Endpoint, MSI 01 > > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s > > <512ns, L1 <64us > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > > Unsupported- > > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- > > MaxPayload 128 bytes, MaxReadReq 4096 bytes > > DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ > > TransPend- > > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, > > Latency L0 <512ns, L1 <64us > > ClockPM+ Suprise- LLActRep- BwNot- > > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > > CommClk- > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ > > DLActive- BWMgmt- ABWMgmt- > > Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4 > > Vector table: BAR=4 offset=00000000 > > PBA: BAR=4 offset=00000800 > > Capabilities: [cc] Vital Product Data <?> > > Capabilities: [100] Advanced Error Reporting <?> > > Capabilities: [140] Virtual Channel <?> > > Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-00-03 > > Kernel driver in use: r8168 > > Kernel modules: r8168, r8169 > > > > > > do you think I am missing something??? > > > > Your help will be appreciated. > > Thank you > > > > > > Please remember to CC the xen-users list! > > According to the NIC''s page: > http://www.realtek.com/products/productsView.aspx?Langid=1&PNid=13&PFid=5&Level=5&Conn=4&ProdID=12 > > Your controller does not have SR-IOV! AFAIK, only some Intel > controllers have SR-IOV: > (Pause at 9:09, SR-IOV is on the right side of that list, not the left side) > http://www.youtube.com/watch?v=hRHsk8Nycdg#t=9m9s > > You can check which controller has SR-IOV by checking its page at > http://ark.intel.com > > Hope this helps > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello,
Replies inline...
On 24 November 2011 03:27, Achala Aryal <arya2595@vandals.uidaho.edu>
wrote:>
> Oh got it.
> So If I want to passthrough this device (which is PCIe not PCI ) then is it
> a same procedure like in PCI passthrough
> hiding it from dom0 and assigning it to domU config flie....
Yes
> also one more since this is pcie device, I can passthrough to more than one
> domU right?
Unfortunately, no. It is still limited to a one-to-one mapping. Only
SR-IOV allows this by presenting virtual devices. Your best bet would
be to either:
1. Obtain more NICs and individually passthrough each one to the VM, or
2. Use emulated NICs and bridge them to your hardware NIC (at the
cost of software processing overhead)
> Thank you
>
>
what is virtual slot things then? and also lets say, I have pci NIC in dom0 and two different PCIe device and are assigned it directly to two different VMs. is there any communication problem between them? like PCI to PCIe device?> From: xieliwei@gmail.com > Date: Thu, 24 Nov 2011 03:35:03 +0800 > To: arya2595@vandals.uidaho.edu > CC: xen-users@lists.xensource.com > Subject: Re: [Xen-users] SR-IOV > > Hello, > Replies inline... > > On 24 November 2011 03:27, Achala Aryal <arya2595@vandals.uidaho.edu> wrote: > > > > Oh got it. > > So If I want to passthrough this device (which is PCIe not PCI ) then is it > > a same procedure like in PCI passthrough > > hiding it from dom0 and assigning it to domU config flie.... > > Yes > > > also one more since this is pcie device, I can passthrough to more than one > > domU right? > > Unfortunately, no. It is still limited to a one-to-one mapping. Only > SR-IOV allows this by presenting virtual devices. Your best bet would > be to either: > 1. Obtain more NICs and individually passthrough each one to the VM, or> 2. Use emulated NICs and bridge them to your hardware NIC (at the > cost of software processing overhead) > > > Thank you > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 27 November 2011 10:23, Achala Aryal <arya2595@vandals.uidaho.edu> wrote:> what is virtual slot things then?I''m assuming that you''re referring to this: http://wiki.xen.org/xenwiki/VTdHowTo#line-199 Recall that PCI passthrough is basically assigning a physical PCI card to a virtual machine. Within the VM are virtual slots, virtual counterparts of the physical PCI slots you plug your cards into. So when you assign a PCI device to a virtual slot, its like slotting the physical PCI card into the VM''s virtual PCI slot. See Slide 24 - 29 of http://www.valinux.co.jp/documents/tech/presentlib/2009/jls/multi-function_a.pdf for some nice graphical representations (although the graphics refer to virtual functions, the idea is very similar).> > and also lets say, > I have pci NIC in dom0 and two different PCIe device and are assigned it > directly to two different VMs. > > > is there any communication problem between them? like PCI to PCIe device? >In general, there should be no problems, except for some device and BIOS quirks. For lower level considerations like overheads and fringe scenarios, you''ll have to ask the xen devs. One thing to note is that, it is likely your PCI slots are provided by a PCIe-to-PCI bridge. If that is so, last I heard, all PCI devices behind the same bridge must be assigned to the same domain: http://www.ibm.com/developerworks/linux/library/l-pci-passthrough/#hypervisor_support
Thank you for the link and clearification. One more thing Is there a command to see memory address of guest domain? And also any command that shows unauthorized access to memory if try to access memory of different domain? I only want is to proof isolation of vm with the use of vt-d. Any idea? Btw I successfully passthrough both pci and pcie device to guest domain. Thank you On Nov 26, 2011, at 11:07 PM, "Liwei" <xieliwei@gmail.com> wrote:> On 27 November 2011 10:23, Achala Aryal <arya2595@vandals.uidaho.edu> wrote: >> what is virtual slot things then? > > I''m assuming that you''re referring to this: > http://wiki.xen.org/xenwiki/VTdHowTo#line-199 > > Recall that PCI passthrough is basically assigning a physical PCI card > to a virtual machine. Within the VM are virtual slots, virtual > counterparts of the physical PCI slots you plug your cards into. So > when you assign a PCI device to a virtual slot, its like slotting the > physical PCI card into the VM''s virtual PCI slot. > > See Slide 24 - 29 of > http://www.valinux.co.jp/documents/tech/presentlib/2009/jls/multi-function_a.pdf > for some nice graphical representations (although the graphics refer > to virtual functions, the idea is very similar). > >> >> and also lets say, >> I have pci NIC in dom0 and two different PCIe device and are assigned it >> directly to two different VMs. >> >> >> is there any communication problem between them? like PCI to PCIe device? >> > > In general, there should be no problems, except for some device and > BIOS quirks. For lower level considerations like overheads and fringe > scenarios, you''ll have to ask the xen devs. > > One thing to note is that, it is likely your PCI slots are provided by > a PCIe-to-PCI bridge. If that is so, last I heard, all PCI devices > behind the same bridge must be assigned to the same domain: > > http://www.ibm.com/developerworks/linux/library/l-pci-passthrough/#hypervisor_support > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users
On 28 November 2011 00:01, Achala Aryal <arya2595@vandals.uidaho.edu> wrote:> Thank you for the link and clearification. > > One more thing Is there a command to see memory address of guest domain?I recall that there''s already some dprintk in the xen sources that let''s you see this at creation time, not sure where though. Try digging through the relevant source files and enable debug printouts for them (assuming the pr_debug macro is used): http://www.kernel.org/doc/local/pr_debug.txt Also, I believe the allocation may not be contiguous, so do keep that in mind. Or can anyone else answer this? Sorry but I am of not much help in this area.> And also any command that shows unauthorized access to memory if try to access memory of different domain?Generally, barring any exploitation circumventing isolation, there is no way for a domU to access memory it is not allocated, except perhaps via DMA. VT-D is supposed to prevent this though. Again, this is out of my comfort zone, so I may be completely wrong.> > I only want is to proof isolation of vm with the use of vt-d. Any idea?Start by taking a look at xen/drivers/passthrough/vtd/iommu.c and see how you may test it.> > Btw I successfully passthrough both pci and pcie device to guest domain. > > Thank youGreat to know! You''re welcome! =)>List, please do correct me. I''m interested in knowing the answers as well.