Xen devel - Jun 2010 - SR-IOV function passthrough to a PV domU

I''m trying to pass through an SR-IOV virtual function to a PV domU.
The pci passthrough seems to work, but the virtual function never shows
up on the pci bus of the domU.  All I see is a message stating:
[  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00

I''m using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and domU
and Xen 4.0.  I''ve recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG
option
set, but I don''t get any additional output when attaching the device.

Is it possible to use pci passthrough with SR-IOV virtual functions to a
PV domU?

I noticed in this thread:
http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
That someone mentioned a specific SR-IOV pcifront driver.
Does anyone know where this driver may be available?

For what it''s worth, I''ve also tried kernel 2.6.34 with the
xen patches
from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
the same results.

Thanks,
Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jun 15, 2010 at 03:15:35PM -0600, Chris Reeves
wrote:
> I''m trying to pass through an SR-IOV virtual function to a PV
domU.
> The pci passthrough seems to work, but the virtual function never shows
> up on the pci bus of the domU.  All I see is a message stating:
> [  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
> 
> I''m using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and
domU
> and Xen 4.0.  I''ve recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG
option
> set, but I don''t get any additional output when attaching the
device.
> 
> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?
Yes. But your Dom0 kernel MUST be the updated PV-OPS version. You need
to get the latest pv-ops kernel. Details at:
http://wiki.xensource.com/xenwiki/XenParavirtOps
> 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?
In the PV-OPS tree.
> 
> For what it''s worth, I''ve also tried kernel 2.6.34 with
the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.
Yeah, no surprise. They don''t have the fixes for Xen PCIBack working
with SR-IOV hardware.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The pciback driver in dom0 needs patching.

See http://xenbits.xensource.com/linux-2.6.18-xen.hg changeset 999, for
example. I would imagine some vendor kernels may be missing this, although I
thought Novell had it. Jan might be able to say (cc''ed).

 -- Keir

On 15/06/2010 22:15, "Chris Reeves" <chrisr@spectralogic.com>
wrote:
> I''m trying to pass through an SR-IOV virtual function to a PV
domU.
> The pci passthrough seems to work, but the virtual function never shows
> up on the pci bus of the domU.  All I see is a message stating:
> [  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
> 
> I''m using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and
domU
> and Xen 4.0.  I''ve recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG
option
> set, but I don''t get any additional output when attaching the
device.
> 
> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?
> 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?
> 
> For what it''s worth, I''ve also tried kernel 2.6.34 with
the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.
> 
> Thanks,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?
Yes, though you need to use a dom0 kernel with the attached pciback patch.

XCP 0.5 has the relevant patch.

Ian

 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?
> 
> For what it''s worth, I''ve also tried kernel 2.6.34 with
the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.
> 
> Thanks,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt
wrote:
> > Is it possible to use pci passthrough with SR-IOV virtual functions to
a
> > PV domU?
> 
> Yes, though you need to use a dom0 kernel with the attached pciback patch.
> 
> XCP 0.5 has the relevant patch.
> 
I wonder if pvops dom0 kernels have this patch? 

-- Pasi
> 
>  
> > I noticed in this thread:
> >
http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> > That someone mentioned a specific SR-IOV pcifront driver.
> > Does anyone know where this driver may be available?
> > 
> > For what it''s worth, I''ve also tried kernel 2.6.34
with the xen patches
> > from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I
get
> > the same results.
> > 
> > Thanks,
> > Chris
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/06/2010 07:44, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt wrote:
>>> Is it possible to use pci passthrough with SR-IOV virtual functions
to a
>>> PV domU?
>> 
>> Yes, though you need to use a dom0 kernel with the attached pciback
patch.
>> 
>> XCP 0.5 has the relevant patch.
>> 
> 
> I wonder if pvops dom0 kernels have this patch?
I think it does, in the .31 branch at least.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 15.06.10 at 23:31, Ian Pratt <Ian.Pratt@eu.citrix.com>
wrote:
>>  Is it possible to use pci passthrough with SR-IOV virtual functions to
a
>> PV domU?
> 
> Yes, though you need to use a dom0 kernel with the attached pciback patch.
That patch is an almost complete revert of c/s 999 - is that really the
right thing to have?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 15.06.10 at 23:29, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:
> The pciback driver in dom0 needs patching.
> 
> See http://xenbits.xensource.com/linux-2.6.18-xen.hg changeset 999, for
> example. I would imagine some vendor kernels may be missing this, although
I
> thought Novell had it. Jan might be able to say (cc''ed).
The 2.6.31 based tree of ours doesn''t have it, but the 2.6.32 and
2.6.34 based ones do.

I would consider adding that c/s to the .31 tree, but only if I knew it
indeed helps (with you saying .34 - though it''s not really clear what
patch level you have, since I understand it''s not a SuSE kernel
you''re
using - doesn''t work for you either it doesn''t seem like it
would).

For the .32 tree we definitely know this works for others, and I have
no reason to believe that it got broken with the update to .34.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/06/2010 08:59, "Jan Beulich" <JBeulich@novell.com> wrote:
>>>> On 15.06.10 at 23:31, Ian Pratt <Ian.Pratt@eu.citrix.com>
wrote:
>>>  Is it possible to use pci passthrough with SR-IOV virtual
functions to a
>>> PV domU?
>> 
>> Yes, though you need to use a dom0 kernel with the attached pciback
patch.
> 
> That patch is an almost complete revert of c/s 999 - is that really the
> right thing to have?
I expect Ian attached my patch to pciback. C/s 999 reverts that and applies
Intel''s alternative patch in its place, since we decided that was a
slightly
better fix. Really either patch would do.

The existence of Intel''s patch was raised by you, Jan, so I assumed you
must
already have the Intel fix in your trees.

 -- Keir
> Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 16.06.10 at 10:37, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:
> The existence of Intel''s patch was raised by you, Jan, so I
assumed you must
> already have the Intel fix in your trees.
Yes, we started having it as a tentative patch around the release of
11.2 (using the .31 kernel). It never got validated there, so it never
got committed to the repo for that tree. It did get validated for .32
(SLE11 SP1), and then when I saw your version of the patch appear
I submitted ours (or really Intel''s) as a replacement. And I think I
even had submitted it quite a while earlier, but it didn''t get applied
for a reason I no longer remember. Or maybe I had just asked Intel
to submit it, and they never did...

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jun 16, 2010 at 08:09:12AM +0100, Keir Fraser
wrote:
> On 16/06/2010 07:44, "Pasi Kärkkäinen" <pasik@iki.fi>
wrote:
> 
> > On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt wrote:
> >>> Is it possible to use pci passthrough with SR-IOV virtual
functions to a
> >>> PV domU?
> >> 
> >> Yes, though you need to use a dom0 kernel with the attached
pciback patch.
> >> 
> >> XCP 0.5 has the relevant patch.
> >> 
> > 
> > I wonder if pvops dom0 kernels have this patch?
> 
> I think it does, in the .31 branch at least.
And .32.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for the help!  I used the patch Ian attached, and I could
successfully pass through the VFs.  I''ll try changeset 999 next time if
that is the preferred solution.

Jan, I can try c/s 999 on my setup sooner if you''d be interested in the
results.

Thanks again,
Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 17/06/2010 01:15, "Chris Reeves" <chrisr@spectralogic.com>
wrote:
> Thanks for the help!  I used the patch Ian attached, and I could
> successfully pass through the VFs.  I''ll try changeset 999 next
time if
> that is the preferred solution.
> 
> Jan, I can try c/s 999 on my setup sooner if you''d be interested
in the
> results.
I think whatever works for you is just fine to carry on with. As Jan says, a
suitable patch is already applied to more recent Novell kernels.

 -- Keir
> Thanks again,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel