Xen users - May 2008 - Simple Query on PCI passthrough I/O

Hi,

I want to perform pass through I/O from my domU. I guess I would need an
additional network card for the same.
Is there any preference on the second network card I should get or any one
is fine? I have a x86_64 machine
with Cent OS 4.4 running. Are there any additional considerations that I
cannot see - Kindly let me know.

Thanks,
Asim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

You only need a nic that your DomU supports (or can with a new module).
since you hide it and pass it through in Dom0, it doesn''t even see
it...

jlc

From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Asim
Sent: Monday, May 19, 2008 12:51 PM
To: xen-users@lists.xensource.com
Subject: [Xen-users] Simple Query on PCI passthrough I/O

Hi,

I want to perform pass through I/O from my domU. I guess I would need an
additional network card for the same.
Is there any preference on the second network card I should get or any one is
fine? I have a x86_64 machine
with Cent OS 4.4 running. Are there any additional considerations that I cannot
see - Kindly let me know.

Thanks,
Asim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 19 May 2008, Joseph L. Casale wrote:
> You only need a nic that your DomU supports (or can with a new module).
> since you hide it and pass it through in Dom0, it doesn''t even see
it...
Indeed.  In principle pretty much any PCI device ought to be possible to pass 
through to a guest.  As long as the guest can support that network card then 
it should work OK.

There are a few PCI devices that have caused people trouble when doing 
passthrough (searching the mailing list archives on xen-devel and xen-users 
may reveal some of these) sometimes due to odd hardware behaviour, sometimes 
due to driver bugs.  In general there''s not a hard-and-fast rule other
than
observing other''s experience.  Plenty of people seem to be happily
using this
feature, though.

I''d suggest you try a NIC (maybe just try with the current NIC in your
host,
initially) and see if you can make it work.  I''d guess that you have a
pretty
high chance of it working once you''ve got the setup right.  You could
then
chose to buy another of the known-good card in your host, or try something 
else.

Other things to be aware of:

* Passing a PCI device to a guest makes that guest as trusted as dom0. 
It''s
just as powerful in terms of being able to snoop data, crash the system, 
etc...

* ... unless you are using VT-d, in which case you can pass the PCI device to 
an HVM guest without that guest being able to stomp all over memory.  Patches 
to support this for PV guests are in development but not done yet.

* Don''t try to xm pause, save or suspend the guest with the PCI card! 
Bad
things may well happen ;-)

* You need to be running a dom0 kernel in the guest with the PCI card.

Cheers,
Mark
> jlc
>
> From: xen-users-bounces@lists.xensource.com
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Asim Sent:
> Monday, May 19, 2008 12:51 PM
> To: xen-users@lists.xensource.com
> Subject: [Xen-users] Simple Query on PCI passthrough I/O
>
> Hi,
>
> I want to perform pass through I/O from my domU. I guess I would need an
> additional network card for the same. Is there any preference on the second
> network card I should get or any one is fine? I have a x86_64 machine with
> Cent OS 4.4 running. Are there any additional considerations that I cannot
> see - Kindly let me know.
>
> Thanks,
> Asim


-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 19, 2008 at 5:40 PM, Mark Williamson <
mark.williamson@cl.cam.ac.uk> wrote:
> On Monday 19 May 2008, Joseph L. Casale wrote:
> > You only need a nic that your DomU supports (or can with a new
module).
> > since you hide it and pass it through in Dom0, it doesn''t
even see it...
>
> Indeed.  In principle pretty much any PCI device ought to be possible to
> pass
> through to a guest.  As long as the guest can support that network card
> then
> it should work OK.
>
> There are a few PCI devices that have caused people trouble when doing
> passthrough (searching the mailing list archives on xen-devel and xen-users
> may reveal some of these) sometimes due to odd hardware behaviour,
> sometimes
> due to driver bugs.  In general there''s not a hard-and-fast rule
other than
> observing other''s experience.  Plenty of people seem to be happily
using
> this
> feature, though.
>
> I''d suggest you try a NIC (maybe just try with the current NIC in
your
> host,
> initially) and see if you can make it work.  I''d guess that you
have a
> pretty
> high chance of it working once you''ve got the setup right.  You
could then
> chose to buy another of the known-good card in your host, or try something
> else.
>
> Other things to be aware of:
>
> * Passing a PCI device to a guest makes that guest as trusted as dom0.
>  It''s
> just as powerful in terms of being able to snoop data, crash the system,
> etc...
>
> * ... unless you are using VT-d, in which case you can pass the PCI device
> to
> an HVM guest without that guest being able to stomp all over memory.
>  Patches
> to support this for PV guests are in development but not done yet.
>
> * Don''t try to xm pause, save or suspend the guest with the PCI
card!  Bad
> things may well happen ;-)
>
> * You need to be running a dom0 kernel in the guest with the PCI card.
>
> Cheers,
> Mark
>
> > jlc
> >
> > From: xen-users-bounces@lists.xensource.com
> > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Asim Sent:
> > Monday, May 19, 2008 12:51 PM
> > To: xen-users@lists.xensource.com
> > Subject: [Xen-users] Simple Query on PCI passthrough I/O
> >
> > Hi,
> >
> > I want to perform pass through I/O from my domU. I guess I would need
an
> > additional network card for the same. Is there any preference on the
> second
> > network card I should get or any one is fine? I have a x86_64 machine
> with
> > Cent OS 4.4 running. Are there any additional considerations that I
> cannot
> > see - Kindly let me know.
> >
> > Thanks,
> > Asim
>
>
>
> --
> Push Me Pull You - Distributed SCM tool (
>
http://www.cl.cam.ac.uk/~maw48/pmpu/<http://www.cl.cam.ac.uk/%7Emaw48/pmpu/>
> )
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

Make sure your pci card is not only supported in linux, but supported in a
xen domU.  I have an e100 ethernet card that works fine in dom0 but could
only work in a domU If I use xen 3.2.  Earlier versions caused memory
allocation errors.

Also, my domU insisted on assigning eth0 to the first pci device it sees
despite the alias in /etc/modprobe so it is best to set the physical nic in
domU as eth0.  It took me a while to figure this out.  The domU kept on
complaining about "device e100 not available, delaying
initialization".

Chris


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks a lot for all your responses - Chris, Mark and Joseph. They really
help me.
A few followup questions from your responses.

> * ... unless you are using VT-d, in which case you can pass the PCI device
> to
> an HVM guest without that guest being able to stomp all over memory.
>  Patches
> to support this for PV guests are in development but not done yet.

I''m running a PV. Does this mean I wont be able to do pass  through
I/O?

>
>
> * Don''t try to xm pause, save or suspend the guest with the PCI
card!  Bad
> things may well happen ;-)
>
> * You need to be running a dom0 kernel in the guest with the PCI card.
Whoa! I did not know this either. Again what are the reasons? Kindly let me
know.

Thanks,
Asim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Thanks a lot for all your responses - Chris, Mark and Joseph. They really
> help me.
> A few followup questions from your responses.
>
> > * ... unless you are using VT-d, in which case you can pass the PCI
> > device to
> > an HVM guest without that guest being able to stomp all over memory.
> >  Patches
> > to support this for PV guests are in development but not done yet.
>
> I''m running a PV. Does this mean I wont be able to do pass 
through I/O?
Under PV you can pass through IO.  It just means that the guest with the 
passed-through card is as trusted as dom0.  i.e. the guest with the PCI card 
can potentially read and write all memory in the machine, take control of 
anything, etc, if controlled by a sufficiently determined attacker.

Dom0 already has this much power.  Giving a domU a PCI device does not give it 
*permission* to do these things, so it''s technically more limited than
dom0.
However, it can (in principle) abuse the PCI device to break the system and 
thereby get these powers.  This is not a problem as long as you trust the 
administrator in the domU and it does not get hacked.  Otherwise, it could 
try to escalate its privileges.

Patches to limit PV guests using VT-d hardware where available have been 
reposted to the mailing list today, so that capability may be in the next Xen 
release.  Not helpful to people without that hardware, sadly.
> > * Don''t try to xm pause, save or suspend the guest with the
PCI card!
> > Bad things may well happen ;-)
You shouldn''t pause a guest with a PCI card (or save or suspend it)
because
you might interrupt it whilst it''s doing something important with the 
hardware.  Potentially this can break your system.  Migrating has the same 
problem but also the guest would be confused to arrive on another machine 
without the PCI device it was using, so it wouldn''t be a good idea
anyhow.

Occasionally people have accidentally suspended a domU (the Xendomains init 
script does this on dom0 shutdown on some systems, for instance - it''s
worth
making sure this won''t happen!) that had a PCI device attached.  This
caused
problems - most recently, I guy had dom0 lose access to the hard disk.  The 
reason was that the guest was leaving an interrupt line masked when it 
shutdown - that interrupt line was shared with dom0''s hard drive
controller.

Best to just assume that any domain with hardware access needs to either be 
running or shut down cleanly.  xm destroy-ing one is possible but probably 
not recommended unless necessary, because similar problems could occur.  This 
was never a problem on my own test machine but your mileage may vary on 
different hardware.
> >
> > * You need to be running a dom0 kernel in the guest with the PCI card.
>
> Whoa! I did not know this either. Again what are the reasons? Kindly let me
> know.
Regarding the kernel version.  You don''t necessarily need to be running
the
same kernel as dom0.  But you do need to be running a dom0-capable kernel 
because domU-only kernels lack hardware support.

Finally, I''ll note that you can''t run a really old kernel in a
domU with PCI
passthrough because the dom0 (aka privileged) interfaces were stabilised 
later on than the domU interfaces, so if your kernel is too old it may not 
work.

Sorry if that all sounds horribly dire.  Plenty of people have been using PCI 
passthrough on all sorts of systems for years now, quite happily.  It''s
not
like it''s a highly dangerous activity.  But it''s one of the
more technical
aspects of Xen to set up and it''s worth you knowing the various issues 
associated with deploying it.

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > > * You need to be running a dom0 kernel in the guest with the PCI
card.
> >
> > Whoa! I did not know this either. Again what are the reasons? Kindly
let > me
> > know.
>
> Regarding the kernel version.  You don''t necessarily need to be
> running the same kernel as dom0.  But you do need to be running
> a dom0-capable kernel because domU-only kernels lack hardware support.
Hmm, I thought that meant the domU kernel had to run the pcifront driver
plus the native driver for the PCI device.  Is that right or have I
missed something?

Is there a way to tell if the domU is actually running pcifront?  My
/boot/config-2.6.18-53.el5xen shows CONFIG_XEN_PCIDEV_FRONTEND=y, but is
there a way to verify that it is actually in the running kernel?

Thanks,
Ed


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > > > * You need to be running a dom0 kernel in the guest with the
PCI
>
> card.
>
> > > Whoa! I did not know this either. Again what are the reasons?
Kindly
>
> let > me
>
> > > know.
> >
> > Regarding the kernel version.  You don''t necessarily need to
be
> > running the same kernel as dom0.  But you do need to be running
> > a dom0-capable kernel because domU-only kernels lack hardware support.
>
> Hmm, I thought that meant the domU kernel had to run the pcifront driver
> plus the native driver for the PCI device.  Is that right or have I
> missed something?
I had in mind that some of the privileged operations (i.e. included in dom0 
kernels not in purely unprivileged domU kernels) were required for a domain 
to support a native device driver correctly and so you''d need a
dom0-enabled
kernel.  I *might* be wrong nowadays but I wouldn''t be surprised if it
were
still the case.  Real device drivers use operations that aren''t
necessary for
a purely unprivileged domU kernel to support, after all.

So in conclusion, it *might* be possible to add pcifront and a native device 
driver to an otherwise pure domU kernel but I wouldn''t necessarily
expect it
to work.  Hence I recommend using a dom0 /capable/ kernel in a domU that owns 
a PCI device.  Not necessarily the same kernel as dom0, although may be 
easiest to set up on some installs.

This distinction is less relevant these days since many distributions ship a 
single combined dom0 and domU kernel for all purposes.
> Is there a way to tell if the domU is actually running pcifront?  My
> /boot/config-2.6.18-53.el5xen shows CONFIG_XEN_PCIDEV_FRONTEND=y, but is
> there a way to verify that it is actually in the running kernel?
Well, if it''s =y then I expect it should be in there.  You can probably
check
this by looking for output in dmesg and by looking for an entry in /sysfs.  I 
don''t have a Xen system handy at this moment to check the exact details
for
you, sorry.

Hope that helps,

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users