Xen devel - Nov 2007 - Passthrough support ?

Hi ,


I  have been trying to understand PCI -passthrough support.  Please 
correct me if I am wrong in my following inferences.

1. Device emulation and pass through are both implemented using split 
drivers.
2. But, in case of emulation drivers of dom0 are used  where as in case 
of passthrough ( as the name suggests ) native drivers in domU are used .
3. dom0 provides a virtual PCI device { an interface for device-OPs and 
status  of this virtual device} to domU and  through associated event 
channel domU makes
   "synchronous" use of this device.

==Queries:

1. What i am really not so sure about is ... passthrough case
    Will there be requirement to map the address space of this PCI  
device in domU ?  Will the page which was being shared  so-far 
{xen_pci_sharedinfo}
     for emulation , be "flipped"  ( transferred ) into domU ? 2. Well
,
Having read the code for linux (dom0,domU) I see that there are split  
device  drivers for PCI. (pci front and PCIback). Which are normally
   communicating over xenbus.  which looks almost like other split 
drivers.  How exactly then passthrough enables use of domU''s drivers ?
3. And if passthrough support isnt provided how will communication 
between pcifront-pciback  be different ? ( I guess netbsd , freebsd do 
not have passthrough support yet )
4. What  restricts other domUs from accessing PCI device given to other 
domU via passrthrough support.


Regards,
Sanket

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I  have been trying to understand PCI -passthrough support.  Please
> correct me if I am wrong in my following inferences.
>
> 1. Device emulation and pass through are both implemented using split
> drivers.
I''m going to be pedantic now ;-)

Device "emulation" is really what we do for fully virtualised (HVM)
guests:
the device models provided by QEmu emulate real world devices in terms of 
their responses to particular port IOs, mmapped IO operations, etc.  This 
isn''t done using a front / back model because the guest is just using
it''s
normal drivers for the "real" devices.

The virtual devices used by the PV drivers are implemented using split 
drivers, though, as you say.

The PCI passthrough support for PV guest is also implemented using a split 
driver that implements the functions of the PCI bus in order to give the 
guest the information it requires to talk to its assigned device(s).  Having 
obtained this information, communication with the device is possible directly 
using IO ports, memory IO regions, and DMA.
> 2. But, in case of emulation drivers of dom0 are used  where as in case
> of passthrough ( as the name suggests ) native drivers in domU are used .
For true emulation (qemu device model), a userspace process in dom0 handles 
modelling a "real" device and then issues IO using normal userspace
APIs.
These get serviced by the dom0 kernel using the normal device driver.

For PV drivers, the frontend driver in the domU kernel issues requests which 
are picked up by the backend driver in dom0''s kernel, which then issues
requests into the IO stack.  Again this uses the normal device driver in dom0 
to talk to the actual device, it''s just that the request is made using
a
kernel-internal API rather than a userspace API (which results in slightly 
different actions being taken).
> 3. dom0 provides a virtual PCI device { an interface for device-OPs and
> status  of this virtual device} to domU and  through associated event
> channel domU makes
>    "synchronous" use of this device.
domU uses this for control plane operations, but for most work it can talk to 
its PCI device directly without going through dom0.
> ==> Queries:
>
> 1. What i am really not so sure about is ... passthrough case
>     Will there be requirement to map the address space of this PCI
> device in domU ?  Will the page which was being shared  so-far
> {xen_pci_sharedinfo}
>      for emulation , be "flipped"  ( transferred ) into domU ?
xen_pci_sharedinfo - is that the page used to talk to the PCI backend from 
pcifront?  If so, then no, that''s just used for dom0-domU
communications.

In order to map the address space of the PCI device directly, the domU is 
given permissions to map the IO memory regions of that device into it''s
page
tables.  I think this is now possible to do using a grant table operation...

It is also given permission to access certain IO port ranges so that it can 
use the device''s port IO interfaces.
>      2. Well , 
> Having read the code for linux (dom0,domU) I see that there are split
> device  drivers for PCI. (pci front and PCIback). Which are normally
>    communicating over xenbus.  which looks almost like other split
> drivers.  How exactly then passthrough enables use of domU''s
drivers ?
The key thing to understand is that the pcifront / pciback is basically just 
used for setup and teardown, not for the actual IO.  The real IO is done 
directly by the domU without going through dom0.  For the block and net 
drivers, *all* IO goes through dom0.
> 3. And if passthrough support isnt provided how will communication
> between pcifront-pciback  be different ? ( I guess netbsd , freebsd do
> not have passthrough support yet )
I''m not entirely clear what you''re asking here, but
I''ll take a stab at it:

if pcifront (in domU) and pciback (in dom0) aren''t available then
passthrough
won''t work.  The dom0 has to support the backend functions of PCI
passthrough
and the domU has to know how to talk to it.  It''s also implicit that
they''re
using the same interface version to talk to each other - I''m not sure
whether
that''s frozen stable or not.

So *if* NetBSD lacks pciback support, it can''t pass PCI devices to
guests that
do.  Similarly, *if* it lacks pcifront support, it can''t have devices
passed
to it.
> 4. What  restricts other domUs from accessing PCI device given to other
> domU via passrthrough support.
There are some restrictions on what can be done in PCI config space to prevent 
a guest fouling things up.  These need to be relaxed for some awkward 
devices, though.

For the device IO itself, domUs are only allowed to map mmio regions and 
access io ports that are relevant to their device.  It''s possible for
these
to overlap with those for other devices, in which case you''re trusting
the
domU to be well behaved.  More crucially, though, giving a domain a device 
with DMA capabilities is equivalent to giving it the ability to subvert the 
entire machine.  DMA can''t be sandboxed on most current hardware, so if
you
give DMA rights to a VM it''s automatically just as trusted as dom0 with
respect to not fooling about with other domains, hardware, etc.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Good brainer. ... I  am geting back to few basics.
I will leave HVM and emulation aside :)

1. When PCI Passthrough support  ''is not enabled'' , how does
domU access
PCI device ?
    I suppose they continue to communicate using pcifront/back split 
drivers and now dom0''s
    drivers are used ( right ? )

2. PCI-passthrough enables domU to own the PCI device. Now dom0 can no 
longer use that device .
    If domU has mapped address space of the required  PCI device , why 
need it talk to dom0 any  further ?
    What kind of hand shake is involved ( setup and tear down as you 
have mentioned ) between domU and
    domU  " in passthrough case" ?

3. DomU discovers all PCI devices through  Xenstore.
 
Mark Williamson wrote:
>> I  have been trying to understand PCI -passthrough support.  Please
>> correct me if I am wrong in my following inferences.
>>
>> 1. Device emulation and pass through are both implemented using split
>> drivers.
>>     
>
> I''m going to be pedantic now ;-)
>
> Device "emulation" is really what we do for fully virtualised
(HVM) guests:
> the device models provided by QEmu emulate real world devices in terms of 
> their responses to particular port IOs, mmapped IO operations, etc.  This 
> isn''t done using a front / back model because the guest is just
using it''s
> normal drivers for the "real" devices.
>
> The virtual devices used by the PV drivers are implemented using split 
> drivers, though, as you say.
>
> The PCI passthrough support for PV guest is also implemented using a split 
> driver that implements the functions of the PCI bus in order to give the 
> guest the information  it 
Which is this information we are referring to here ?
>  requires to talk to its assigned device(s). 
How are devices "assigned"  to domU ?  I am specifically talking about
late binding.
>  Having 
> obtained this information, communication with the device is possible
directly
> using IO ports, memory IO regions, and DMA.
>
>   
>> 2. But, in case of emulation drivers of dom0 are used  where as in case
>> of passthrough ( as the name suggests ) native drivers in domU are used
.
>>     
>
> For true emulation (qemu device model), a userspace process in dom0 handles
> modelling a "real" device and then issues IO using normal
userspace APIs.
> These get serviced by the dom0 kernel using the normal device driver.
>
> For PV drivers, the frontend driver in the domU kernel issues requests
which
> are picked up by the backend driver in dom0''s kernel, which then
issues
> requests into the IO stack.  Again this uses the normal device driver in
dom0
> to talk to the actual device, it''s just that the request is made
using a
> kernel-internal API rather than a userspace API (which results in slightly 
> different actions being taken).
>
>   
>> 3. dom0 provides a virtual PCI device { an interface for device-OPs and
>> status  of this virtual device} to domU and  through associated event
>> channel domU makes
>>    "synchronous" use of this device.
>>     
>
> domU uses this for control plane operations, but for most work it can talk
to
> its PCI device directly without going through dom0.
>
>   
>> ==>> Queries:
>>
>> 1. What i am really not so sure about is ... passthrough case
>>     Will there be requirement to map the address space of this PCI
>> device in domU ?  Will the page which was being shared  so-far
>> {xen_pci_sharedinfo}
>>      for emulation , be "flipped"  ( transferred ) into domU
?
>>     
>
> xen_pci_sharedinfo - is that the page used to talk to the PCI backend from 
> pcifront?  If so, then no, that''s just used for dom0-domU
communications.
>   
Well As I read in the xen0linux code , xen_pci_sharedinfo contains  
xen_pci_op where front end specifies which
operation is to be done on which pci device. And then it keeps checking 
the status of this device through another flag in
xen_pci_sharedinfo. I infer here that dom0''s drivers are being used.
Can
you point me towards a code path where domU actually
uses its own drivers having owned the PCI device ?
> In order to map the address space of the PCI device directly, the domU is 
> given permissions to map the IO memory regions of that device into
it''s page
> tables.  I think this is now possible to do using a grant table
operation...
>
> It is also given permission to access certain IO port ranges so that it can
> use the device''s port IO interfaces.
>
>   
>>      2. Well , 
>> Having read the code for linux (dom0,domU) I see that there are split
>> device  drivers for PCI. (pci front and PCIback). Which are normally
>>    communicating over xenbus.  which looks almost like other split
>> drivers.  How exactly then passthrough enables use of domU''s
drivers ?
>>     
>
> The key thing to understand is that the pcifront / pciback is basically
just
> used for setup and teardown, not for the actual IO.  The real IO is done 
> directly by the domU without going through dom0.  For the block and net 
> drivers, *all* IO goes through dom0.
>
>   
>> 3. And if passthrough support isnt provided how will communication
>> between pcifront-pciback  be different ? ( I guess netbsd , freebsd do
>> not have passthrough support yet )
>>     
>
> I''m not entirely clear what you''re asking here, but
I''ll take a stab at it:
>
> if pcifront (in domU) and pciback (in dom0) aren''t available then
passthrough
> won''t work.  The dom0 has to support the backend functions of PCI
passthrough
> and the domU has to know how to talk to it.  It''s also implicit
that they''re
> using the same interface version to talk to each other - I''m not
sure whether
> that''s frozen stable or not.
>
> So *if* NetBSD lacks pciback support, it can''t pass PCI devices to
guests that
> do.  Similarly, *if* it lacks pcifront support, it can''t have
devices passed
> to it.
>
>   
>> 4. What  restricts other domUs from accessing PCI device given to other
>> domU via passrthrough support.
>>     
>
> There are some restrictions on what can be done in PCI config space to
prevent
> a guest fouling things up.  These need to be relaxed for some awkward 
> devices, though.
>
> For the device IO itself, domUs are only allowed to map mmio regions and 
> access io ports that are relevant to their device.  It''s possible
for these
> to overlap with those for other devices, in which case you''re
trusting the
> domU to be well behaved.  More crucially, though, giving a domain a device 
> with DMA capabilities is equivalent to giving it the ability to subvert the
> entire machine.  DMA can''t be sandboxed on most current hardware,
so if you
> give DMA rights to a VM it''s automatically just as trusted as dom0
with
> respect to not fooling about with other domains, hardware, etc.
>
> Cheers,
> Mark
>
>   
Thanks,
Sanket

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Good brainer. ... I  am geting back to few basics.
I will leave HVM and emulation aside :)

1. When PCI Passthrough support  ''is not enabled'' , how does
domU access
PCI device ?
   I suppose they continue to communicate using pcifront/back split 
drivers and now dom0''s
   drivers are used ( right ? )

2. PCI-passthrough enables domU to own the PCI device. Now dom0 can no 
longer use that device .
   If domU has mapped address space of the required  PCI device , why 
need it talk to dom0 any  further ?
   What kind of hand shake is involved ( setup and tear down as you have 
mentioned ) between domU and
   domU  " in passthrough case" ?

3. DomU discovers all PCI devices through  Xenstore.

Mark Williamson wrote:
>> I  have been trying to understand PCI -passthrough support.  Please
>> correct me if I am wrong in my following inferences.
>>
>> 1. Device emulation and pass through are both implemented using split
>> drivers.
>>     
>
> I''m going to be pedantic now ;-)
>
> Device "emulation" is really what we do for fully virtualised
(HVM)
> guests: the device models provided by QEmu emulate real world devices 
> in terms of their responses to particular port IOs, mmapped IO 
> operations, etc.  This isn''t done using a front / back model
because
> the guest is just using it''s normal drivers for the
"real" devices.
>
> The virtual devices used by the PV drivers are implemented using split 
> drivers, though, as you say.
>
> The PCI passthrough support for PV guest is also implemented using a 
> split driver that implements the functions of the PCI bus in order to 
> give the guest the information  it 
Which is this information we are referring to here ?
>  requires to talk to its assigned device(s). 
How are devices "assigned"  to domU ?  I am specifically talking about
late binding.
>  Having obtained this information, communication with the device is 
> possible directly using IO ports, memory IO regions, and DMA.
>
>  
>> 2. But, in case of emulation drivers of dom0 are used  where as in case
>> of passthrough ( as the name suggests ) native drivers in domU are 
>> used .
>>     
>
> For true emulation (qemu device model), a userspace process in dom0 
> handles modelling a "real" device and then issues IO using normal
> userspace APIs.  These get serviced by the dom0 kernel using the 
> normal device driver.
>
> For PV drivers, the frontend driver in the domU kernel issues requests 
> which are picked up by the backend driver in dom0''s kernel, which
then
> issues requests into the IO stack.  Again this uses the normal device 
> driver in dom0 to talk to the actual device, it''s just that the 
> request is made using a kernel-internal API rather than a userspace 
> API (which results in slightly different actions being taken).
>
>  
>> 3. dom0 provides a virtual PCI device { an interface for device-OPs and
>> status  of this virtual device} to domU and  through associated event
>> channel domU makes
>>    "synchronous" use of this device.
>>     
>
> domU uses this for control plane operations, but for most work it can 
> talk to its PCI device directly without going through dom0.
>
>  
>> ==>> Queries:
>>
>> 1. What i am really not so sure about is ... passthrough case
>>     Will there be requirement to map the address space of this PCI
>> device in domU ?  Will the page which was being shared  so-far
>> {xen_pci_sharedinfo}
>>      for emulation , be "flipped"  ( transferred ) into domU
?
>>     
>
> xen_pci_sharedinfo - is that the page used to talk to the PCI backend 
> from pcifront?  If so, then no, that''s just used for dom0-domU 
> communications.
>   
Well As I read in the xen0linux code , xen_pci_sharedinfo contains  
xen_pci_op where front end specifies which
operation is to be done on which pci device. And then it keeps checking 
the status of this device through another flag in
xen_pci_sharedinfo. I infer here that dom0''s drivers are being used.
Can
you point me towards a code path where domU actually
uses its own drivers having owned the PCI device ?
> In order to map the address space of the PCI device directly, the domU 
> is given permissions to map the IO memory regions of that device into 
> it''s page tables.  I think this is now possible to do using a
grant
> table operation...
>
> It is also given permission to access certain IO port ranges so that 
> it can use the device''s port IO interfaces.
>
>  
>>      2. Well , Having read the code for linux (dom0,domU) I see that 
>> there are split
>> device  drivers for PCI. (pci front and PCIback). Which are normally
>>    communicating over xenbus.  which looks almost like other split
>> drivers.  How exactly then passthrough enables use of domU''s
drivers ?
>>     
>
> The key thing to understand is that the pcifront / pciback is 
> basically just used for setup and teardown, not for the actual IO.  
> The real IO is done directly by the domU without going through dom0.  
> For the block and net drivers, *all* IO goes through dom0.
>
>  
>> 3. And if passthrough support isnt provided how will communication
>> between pcifront-pciback  be different ? ( I guess netbsd , freebsd do
>> not have passthrough support yet )
>>     
>
> I''m not entirely clear what you''re asking here, but
I''ll take a stab
> at it:
>
> if pcifront (in domU) and pciback (in dom0) aren''t available then 
> passthrough won''t work.  The dom0 has to support the backend
functions
> of PCI passthrough and the domU has to know how to talk to it. 
It''s
> also implicit that they''re using the same interface version to
talk to
> each other - I''m not sure whether that''s frozen stable or
not.
>
> So *if* NetBSD lacks pciback support, it can''t pass PCI devices to
> guests that do.  Similarly, *if* it lacks pcifront support, it
can''t
> have devices passed to it.
>
>  
>> 4. What  restricts other domUs from accessing PCI device given to other
>> domU via passrthrough support.
>>     
>
> There are some restrictions on what can be done in PCI config space to 
> prevent a guest fouling things up.  These need to be relaxed for some 
> awkward devices, though.
>
> For the device IO itself, domUs are only allowed to map mmio regions 
> and access io ports that are relevant to their device.  It''s
possible
> for these to overlap with those for other devices, in which case 
> you''re trusting the domU to be well behaved.  More crucially,
though,
> giving a domain a device with DMA capabilities is equivalent to giving 
> it the ability to subvert the entire machine.  DMA can''t be
sandboxed
> on most current hardware, so if you give DMA rights to a VM it''s 
> automatically just as trusted as dom0 with respect to not fooling 
> about with other domains, hardware, etc.
>
> Cheers,
> Mark
>
>   
Thanks,
Sanket

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 1. When PCI Passthrough support  ''is not enabled'' , how
does domU access
> PCI device ?
>     I suppose they continue to communicate using pcifront/back split
> drivers and now dom0''s
>     drivers are used ( right ? )
If you don''t have PCI passthrough (as provided by pcifront/back) then
the domU
can''t really have direct access to a PCI device.  The only accesses
it''ll be
able to make are by using the virtual block / net interfaces to request dom0 
do some IO for it.
> 2. PCI-passthrough enables domU to own the PCI device. Now dom0 can no
> longer use that device .
Yep.
>     If domU has mapped address space of the required  PCI device , why
> need it talk to dom0 any  further ?
DomU owns the PCI device, but dom0 owns the PCI *bus* hardware.  To query (and 
potentially set) information in "PCI configuration space", the domU
will have
to talk to dom0 somehow, because that information comes from the PCI bus 
hardware, not from the device itself.  The PCI config space information 
includes the location of the mmio and port io regions used by the device, so 
without this communication the domU wouldn''t know how to talk to its
device.

Come to that, the PCI config space information is needed to determine what the 
device *is* so the domU knows what driver to use to talk to it ;-)

The PCI bus control hardware can only be owned by dom0, so it is necessary to 
indirect these operations through dom0.  pciback doesn''t necessarily
pass
through config space data unmodified to pcifront (there used to be a number 
of different options on how it passed stuff through) and it doesn''t 
necessarily allow pcifront to alter config space (for safety reasons).  But 
in the end it provides enough of the functionality of PCI config space to 
enable the domU to find and identify the PCI device(s) it owns, and load the 
appropriate drivers.
>     What kind of hand shake is involved ( setup and tear down as you
> have mentioned ) between domU and
>     domU  " in passthrough case" ?
Well, the operations on PCI config space will go through dom0.  That''s
the
main thing I can think of right now, but there may be other operations too.
> 3. DomU discovers all PCI devices through  Xenstore.
I''m not actually sure on this point, but you should be able to figure
it out
with a scan through the code.  Post here if you have trouble understanding 
sections!
> >  requires to talk to its assigned device(s).
>
> How are devices "assigned"  to domU ?  I am specifically talking
about
> late binding.
pciback needs to "claim" the device in dom0 so that no
"real" device drivers
in dom0 will try to use it.  We don''t want the domains fighting over it
;-)
After dom0 boot time, you can move devices to the control of pciback by 
writing values to sysfs to tell any current driver to release its hold on the 
device, then tell pciback to grab it.

Once pciback has grabbed a device, the rest of dom0 Linux doesn''t know
that a
domU is using the device.  As far as Linux is concerned, the device is owned 
by pciback.

To give the domain access to the device, dom0 needs to issue some hypercalls 
giving the domU the rights to map the appropriate mmio regions, and access 
the appropriate IO regions.  It''ll also need to have setup a connection
between pcifront and pciback.  I guess it might stick something in Xenstore - 
maybe you can see what happens in the code?  Some of this will be done in 
pciback, some is probably administered in the tools code.

I''m sorry to be a bit vague.  It''s a while since I looked at
this code, so
please bare in mind that I''m a little rusty ;-)

Cheers,
Mark

> >  Having
> > obtained this information, communication with the device is possible
> > directly using IO ports, memory IO regions, and DMA.
> >
> >> 2. But, in case of emulation drivers of dom0 are used  where as in
case
> >> of passthrough ( as the name suggests ) native drivers in domU are
used
> >> .
> >
> > For true emulation (qemu device model), a userspace process in dom0
> > handles modelling a "real" device and then issues IO using
normal
> > userspace APIs. These get serviced by the dom0 kernel using the normal
> > device driver.
> >
> > For PV drivers, the frontend driver in the domU kernel issues requests
> > which are picked up by the backend driver in dom0''s kernel,
which then
> > issues requests into the IO stack.  Again this uses the normal device
> > driver in dom0 to talk to the actual device, it''s just that
the request
> > is made using a kernel-internal API rather than a userspace API (which
> > results in slightly different actions being taken).
> >
> >> 3. dom0 provides a virtual PCI device { an interface for
device-OPs and
> >> status  of this virtual device} to domU and  through associated
event
> >> channel domU makes
> >>    "synchronous" use of this device.
> >
> > domU uses this for control plane operations, but for most work it can
> > talk to its PCI device directly without going through dom0.
> >
> >> ==> >> Queries:
> >>
> >> 1. What i am really not so sure about is ... passthrough case
> >>     Will there be requirement to map the address space of this PCI
> >> device in domU ?  Will the page which was being shared  so-far
> >> {xen_pci_sharedinfo}
> >>      for emulation , be "flipped"  ( transferred ) into
domU ?
> >
> > xen_pci_sharedinfo - is that the page used to talk to the PCI backend
> > from pcifront?  If so, then no, that''s just used for
dom0-domU
> > communications.
>
> Well As I read in the xen0linux code , xen_pci_sharedinfo contains
> xen_pci_op where front end specifies which
> operation is to be done on which pci device. And then it keeps checking
> the status of this device through another flag in
> xen_pci_sharedinfo. I infer here that dom0''s drivers are being
used. Can
> you point me towards a code path where domU actually
> uses its own drivers having owned the PCI device ?
>
> > In order to map the address space of the PCI device directly, the domU
is
> > given permissions to map the IO memory regions of that device into
it''s
> > page tables.  I think this is now possible to do using a grant table
> > operation...
> >
> > It is also given permission to access certain IO port ranges so that
it
> > can use the device''s port IO interfaces.
> >
> >>      2. Well ,
> >> Having read the code for linux (dom0,domU) I see that there are
split
> >> device  drivers for PCI. (pci front and PCIback). Which are
normally
> >>    communicating over xenbus.  which looks almost like other split
> >> drivers.  How exactly then passthrough enables use of
domU''s drivers ?
> >
> > The key thing to understand is that the pcifront / pciback is
basically
> > just used for setup and teardown, not for the actual IO.  The real IO
is
> > done directly by the domU without going through dom0.  For the block
and
> > net drivers, *all* IO goes through dom0.
> >
> >> 3. And if passthrough support isnt provided how will communication
> >> between pcifront-pciback  be different ? ( I guess netbsd ,
freebsd do
> >> not have passthrough support yet )
> >
> > I''m not entirely clear what you''re asking here, but
I''ll take a stab at
> > it:
> >
> > if pcifront (in domU) and pciback (in dom0) aren''t available
then
> > passthrough won''t work.  The dom0 has to support the backend
functions of
> > PCI passthrough and the domU has to know how to talk to it. 
It''s also
> > implicit that they''re using the same interface version to
talk to each
> > other - I''m not sure whether that''s frozen stable or
not.
> >
> > So *if* NetBSD lacks pciback support, it can''t pass PCI
devices to guests
> > that do.  Similarly, *if* it lacks pcifront support, it can''t
have
> > devices passed to it.
> >
> >> 4. What  restricts other domUs from accessing PCI device given to
other
> >> domU via passrthrough support.
> >
> > There are some restrictions on what can be done in PCI config space to
> > prevent a guest fouling things up.  These need to be relaxed for some
> > awkward devices, though.
> >
> > For the device IO itself, domUs are only allowed to map mmio regions
and
> > access io ports that are relevant to their device.  It''s
possible for
> > these to overlap with those for other devices, in which case
you''re
> > trusting the domU to be well behaved.  More crucially, though, giving
a
> > domain a device with DMA capabilities is equivalent to giving it the
> > ability to subvert the entire machine.  DMA can''t be
sandboxed on most
> > current hardware, so if you give DMA rights to a VM it''s
automatically
> > just as trusted as dom0 with respect to not fooling about with other
> > domains, hardware, etc.
> >
> > Cheers,
> > Mark
>
> Thanks,
> Sanket
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

We also mentioned about xen_pci_sharedinfo in previous mails
 Well As I read in the linux code , xen_pci_sharedinfo contains xen_pci_op where
front end specifies which
 operation is to be done on which pci device. And then it keeps checking the
status of this device through another flag in
 xen_pci_sharedinfo. I infer here that dom0''s drivers are being used.
Can you point me towards a code path where domU actually
 uses its own drivers having owned the PCI device ? 
 
Mark Williamson wrote:
>> 1. When PCI Passthrough support  ''is not enabled'' ,
how does domU access
>> PCI device ?
>>     I suppose they continue to communicate using pcifront/back split
>> drivers and now dom0''s
>>     drivers are used ( right ? )
>>     
>
> If you don''t have PCI passthrough (as provided by pcifront/back)
then the domU
> can''t really have direct access to a PCI device.  The only
accesses it''ll be
> able to make are by using the virtual block / net interfaces to request
dom0
> do some IO for it.
>
>   
>> 2. PCI-passthrough enables domU to own the PCI device. Now dom0 can no
>> longer use that device .
>>     
>
> Yep.
>
>   
>>     If domU has mapped address space of the required  PCI device , why
>> need it talk to dom0 any  further ?
>>     
>
> DomU owns the PCI device, but dom0 owns the PCI *bus* hardware.  To query
(and
> potentially set) information in "PCI configuration space", the
domU will have
> to talk to dom0 somehow, because that information comes from the PCI bus 
> hardware, not from the device itself.  The PCI config space information 
> includes the location of the mmio and port io regions used by the device,
so
> without this communication the domU wouldn''t know how to talk to
its device.
>
> Come to that, the PCI config space information is needed to determine what
the
> device *is* so the domU knows what driver to use to talk to it ;-)
>
> The PCI bus control hardware can only be owned by dom0, so it is necessary
to
> indirect these operations through dom0.  pciback doesn''t
necessarily pass
> through config space data unmodified to pcifront (there used to be a number
> of different options on how it passed stuff through) and it
doesn''t
> necessarily allow pcifront to alter config space (for safety reasons).  But
> in the end it provides enough of the functionality of PCI config space to 
> enable the domU to find and identify the PCI device(s) it owns, and load
the
> appropriate drivers.
>
>   
>>     What kind of hand shake is involved ( setup and tear down as you
>> have mentioned ) between domU and
>>     domU  " in passthrough case" ?
>>     
>
> Well, the operations on PCI config space will go through dom0. 
That''s the
> main thing I can think of right now, but there may be other operations too.
>
>   
>> 3. DomU discovers all PCI devices through  Xenstore.
>>     
>
> I''m not actually sure on this point, but you should be able to
figure it out
> with a scan through the code.  Post here if you have trouble understanding 
> sections!
>
>   
>>>  requires to talk to its assigned device(s).
>>>       
>> How are devices "assigned"  to domU ?  I am specifically
talking about
>> late binding.
>>     
>
> pciback needs to "claim" the device in dom0 so that no
"real" device drivers
> in dom0 will try to use it.  We don''t want the domains fighting
over it ;-)
> After dom0 boot time, you can move devices to the control of pciback by 
> writing values to sysfs to tell any current driver to release its hold on
the
> device, then tell pciback to grab it.
>
> Once pciback has grabbed a device, the rest of dom0 Linux doesn''t
know that a
> domU is using the device.  As far as Linux is concerned, the device is
owned
> by pciback.
>
> To give the domain access to the device, dom0 needs to issue some
hypercalls
> giving the domU the rights to map the appropriate mmio regions, and access 
> the appropriate IO regions.  It''ll also need to have setup a
connection
> between pcifront and pciback.  I guess it might stick something in Xenstore
-
> maybe you can see what happens in the code?  Some of this will be done in 
> pciback, some is probably administered in the tools code.
>
> I''m sorry to be a bit vague.  It''s a while since I looked
at this code, so
> please bare in mind that I''m a little rusty ;-)
>
> Cheers,
> Mark
>
>
>   
>>>  Having
>>> obtained this information, communication with the device is
possible
>>> directly using IO ports, memory IO regions, and DMA.
>>>
>>>       
>>>> 2. But, in case of emulation drivers of dom0 are used  where as
in case
>>>> of passthrough ( as the name suggests ) native drivers in domU
are used
>>>> .
>>>>         
>>> For true emulation (qemu device model), a userspace process in dom0
>>> handles modelling a "real" device and then issues IO
using normal
>>> userspace APIs. These get serviced by the dom0 kernel using the
normal
>>> device driver.
>>>
>>> For PV drivers, the frontend driver in the domU kernel issues
requests
>>> which are picked up by the backend driver in dom0''s
kernel, which then
>>> issues requests into the IO stack.  Again this uses the normal
device
>>> driver in dom0 to talk to the actual device, it''s just
that the request
>>> is made using a kernel-internal API rather than a userspace API
(which
>>> results in slightly different actions being taken).
>>>
>>>       
>>>> 3. dom0 provides a virtual PCI device { an interface for
device-OPs and
>>>> status  of this virtual device} to domU and  through associated
event
>>>> channel domU makes
>>>>    "synchronous" use of this device.
>>>>         
>>> domU uses this for control plane operations, but for most work it
can
>>> talk to its PCI device directly without going through dom0.
>>>
>>>       
>>>> ==>>>> Queries:
>>>>
>>>> 1. What i am really not so sure about is ... passthrough case
>>>>     Will there be requirement to map the address space of this
PCI
>>>> device in domU ?  Will the page which was being shared  so-far
>>>> {xen_pci_sharedinfo}
>>>>      for emulation , be "flipped"  ( transferred )
into domU ?
>>>>         
>>> xen_pci_sharedinfo - is that the page used to talk to the PCI
backend
>>> from pcifront?  If so, then no, that''s just used for
dom0-domU
>>> communications.
>>>       
>> Well As I read in the xen0linux code , xen_pci_sharedinfo contains
>> xen_pci_op where front end specifies which
>> operation is to be done on which pci device. And then it keeps checking
>> the status of this device through another flag in
>> xen_pci_sharedinfo. I infer here that dom0''s drivers are being
used. Can
>> you point me towards a code path where domU actually
>> uses its own drivers having owned the PCI device ?
>>
>>     
>>> In order to map the address space of the PCI device directly, the
domU is
>>> given permissions to map the IO memory regions of that device into
it''s
>>> page tables.  I think this is now possible to do using a grant
table
>>> operation...
>>>
>>> It is also given permission to access certain IO port ranges so
that it
>>> can use the device''s port IO interfaces.
>>>
>>>       
>>>>      2. Well ,
>>>> Having read the code for linux (dom0,domU) I see that there are
split
>>>> device  drivers for PCI. (pci front and PCIback). Which are
normally
>>>>    communicating over xenbus.  which looks almost like other
split
>>>> drivers.  How exactly then passthrough enables use of
domU''s drivers ?
>>>>         
>>> The key thing to understand is that the pcifront / pciback is
basically
>>> just used for setup and teardown, not for the actual IO.  The real
IO is
>>> done directly by the domU without going through dom0.  For the
block and
>>> net drivers, *all* IO goes through dom0.
>>>
>>>       
>>>> 3. And if passthrough support isnt provided how will
communication
>>>> between pcifront-pciback  be different ? ( I guess netbsd ,
freebsd do
>>>> not have passthrough support yet )
>>>>         
>>> I''m not entirely clear what you''re asking here,
but I''ll take a stab at
>>> it:
>>>
>>> if pcifront (in domU) and pciback (in dom0) aren''t
available then
>>> passthrough won''t work.  The dom0 has to support the
backend functions of
>>> PCI passthrough and the domU has to know how to talk to it. 
It''s also
>>> implicit that they''re using the same interface version to
talk to each
>>> other - I''m not sure whether that''s frozen stable
or not.
>>>
>>> So *if* NetBSD lacks pciback support, it can''t pass PCI
devices to guests
>>> that do.  Similarly, *if* it lacks pcifront support, it
can''t have
>>> devices passed to it.
>>>
>>>       
>>>> 4. What  restricts other domUs from accessing PCI device given
to other
>>>> domU via passrthrough support.
>>>>         
>>> There are some restrictions on what can be done in PCI config space
to
>>> prevent a guest fouling things up.  These need to be relaxed for
some
>>> awkward devices, though.
>>>
>>> For the device IO itself, domUs are only allowed to map mmio
regions and
>>> access io ports that are relevant to their device.  It''s
possible for
>>> these to overlap with those for other devices, in which case
you''re
>>> trusting the domU to be well behaved.  More crucially, though,
giving a
>>> domain a device with DMA capabilities is equivalent to giving it
the
>>> ability to subvert the entire machine.  DMA can''t be
sandboxed on most
>>> current hardware, so if you give DMA rights to a VM it''s
automatically
>>> just as trusted as dom0 with respect to not fooling about with
other
>>> domains, hardware, etc.
>>>
>>> Cheers,
>>> Mark
>>>       
>> Thanks,
>> Sanket
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>     
>
>
>
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>  We also mentioned about xen_pci_sharedinfo in previous mails
>  Well As I read in the linux code , xen_pci_sharedinfo contains xen_pci_op
> where front end specifies which operation is to be done on which pci
> device. And then it keeps checking the status of this device through
> another flag in xen_pci_sharedinfo. I infer here that dom0''s
drivers are
> being used.
Yes, but not dom0''s drivers /for that device/.  Config space ops
submitted
through that interface will use dom0''s drivers for the PCI /bus/, but
that''s
independent of the /specific device/ being used.

Remember there''s two levels in a PCI subsystem: the driver for the bus
(which
will tell you what''s plugged into the bus and how to configure it) and
the
driver for the device (which relies on the driver for the bus to help it talk 
to the device)

pcifront is a PCI bus driver.  The PCI core and the PCI device drivers in the 
domU talk to it in order to do some of their work.  pciback is a service that 
runs in dom0 and lets pcifront instances exchange information with the /real/ 
PCI bus driver.
> Can you point me towards a code path where domU actually uses 
> its own drivers having owned the PCI device ?
pcifront uses pciback to query what PCI devices are available.  pcifront 
appears to the domU as if it''s a real PCI bus, but actually
it''s talking to
the /real/ host PCI bus via bciback.  Config space operations go through the 
PCI bus driver, so in the domU any config space operations go through 
pcifront, to pciback and from there to the real hardware (if appropriate).

Once the domU has got the details of which devices are available (this process 
is done by Linux''s generic PCI probing code, based on information it
gets
from pcifront) it''ll load the driver for the device it''s been
given.  This is
basically just the standard Linux device driver - same as you''d use on
native
Linux - buy compiled against the XenLinux kernel.

The device driver may do a number of things:

1) it might access PCI config space.  It does this through a generic kernel 
API, and the requests are fed into pcifront and satisfied by pciback.  The 
device driver code is the same path as would be run on a native install, 
except that it''s serviced by pcifront not a real PCI bus.

2) it might want to access IO memory regions or IO ports.  It can do this 
directly using the normal APIs (e.g. inb / outb for port IO) etc that Linux 
exports to drivers.  These work *directly*, without going through dom0.

3) it might do DMA to the device.  This will likely be set up using commands 
written to IO memory regions / ports.  It uses standard Linux APIs to set up 
the DMA-able memory; XenLinux includes Xen-aware functions to make sure the 
memory addressing works right (this is the same as in dom0).  The normal 
driver path can be followed here without dom0 interaction.

If the device receives interrupts, I believe they currently come direct from 
Xen without dom0 being involved.

Does that help clarify things?

Cheers,
Mark
> Mark Williamson wrote:
> >> 1. When PCI Passthrough support  ''is not
enabled'' , how does domU access
> >> PCI device ?
> >>     I suppose they continue to communicate using pcifront/back
split
> >> drivers and now dom0''s
> >>     drivers are used ( right ? )
> >
> > If you don''t have PCI passthrough (as provided by
pcifront/back) then the
> > domU can''t really have direct access to a PCI device.  The
only accesses
> > it''ll be able to make are by using the virtual block / net
interfaces to
> > request dom0 do some IO for it.
> >
> >> 2. PCI-passthrough enables domU to own the PCI device. Now dom0
can no
> >> longer use that device .
> >
> > Yep.
> >
> >>     If domU has mapped address space of the required  PCI device ,
why
> >> need it talk to dom0 any  further ?
> >
> > DomU owns the PCI device, but dom0 owns the PCI *bus* hardware.  To
query
> > (and potentially set) information in "PCI configuration
space", the domU
> > will have to talk to dom0 somehow, because that information comes from
> > the PCI bus hardware, not from the device itself.  The PCI config
space
> > information includes the location of the mmio and port io regions used
by
> > the device, so without this communication the domU wouldn''t
know how to
> > talk to its device.
> >
> > Come to that, the PCI config space information is needed to determine
> > what the device *is* so the domU knows what driver to use to talk to
it
> > ;-)
> >
> > The PCI bus control hardware can only be owned by dom0, so it is
> > necessary to indirect these operations through dom0.  pciback
doesn''t
> > necessarily pass through config space data unmodified to pcifront
(there
> > used to be a number of different options on how it passed stuff
through)
> > and it doesn''t necessarily allow pcifront to alter config
space (for
> > safety reasons).  But in the end it provides enough of the
functionality
> > of PCI config space to enable the domU to find and identify the PCI
> > device(s) it owns, and load the appropriate drivers.
> >
> >>     What kind of hand shake is involved ( setup and tear down as
you
> >> have mentioned ) between domU and
> >>     domU  " in passthrough case" ?
> >
> > Well, the operations on PCI config space will go through dom0. 
That''s
> > the main thing I can think of right now, but there may be other
> > operations too.
> >
> >> 3. DomU discovers all PCI devices through  Xenstore.
> >
> > I''m not actually sure on this point, but you should be able
to figure it
> > out with a scan through the code.  Post here if you have trouble
> > understanding sections!
> >
> >>>  requires to talk to its assigned device(s).
> >>
> >> How are devices "assigned"  to domU ?  I am specifically
talking about
> >> late binding.
> >
> > pciback needs to "claim" the device in dom0 so that no
"real" device
> > drivers in dom0 will try to use it.  We don''t want the
domains fighting
> > over it ;-) After dom0 boot time, you can move devices to the control
of
> > pciback by writing values to sysfs to tell any current driver to
release
> > its hold on the device, then tell pciback to grab it.
> >
> > Once pciback has grabbed a device, the rest of dom0 Linux
doesn''t know
> > that a domU is using the device.  As far as Linux is concerned, the
> > device is owned by pciback.
> >
> > To give the domain access to the device, dom0 needs to issue some
> > hypercalls giving the domU the rights to map the appropriate mmio
> > regions, and access the appropriate IO regions.  It''ll also
need to have
> > setup a connection between pcifront and pciback.  I guess it might
stick
> > something in Xenstore - maybe you can see what happens in the code? 
Some
> > of this will be done in pciback, some is probably administered in the
> > tools code.
> >
> > I''m sorry to be a bit vague.  It''s a while since I
looked at this code,
> > so please bare in mind that I''m a little rusty ;-)
> >
> > Cheers,
> > Mark
> >
> >>>  Having
> >>> obtained this information, communication with the device is
possible
> >>> directly using IO ports, memory IO regions, and DMA.
> >>>
> >>>> 2. But, in case of emulation drivers of dom0 are used 
where as in
> >>>> case of passthrough ( as the name suggests ) native
drivers in domU
> >>>> are used .
> >>>
> >>> For true emulation (qemu device model), a userspace process in
dom0
> >>> handles modelling a "real" device and then issues IO
using normal
> >>> userspace APIs. These get serviced by the dom0 kernel using
the normal
> >>> device driver.
> >>>
> >>> For PV drivers, the frontend driver in the domU kernel issues
requests
> >>> which are picked up by the backend driver in dom0''s
kernel, which then
> >>> issues requests into the IO stack.  Again this uses the normal
device
> >>> driver in dom0 to talk to the actual device, it''s
just that the request
> >>> is made using a kernel-internal API rather than a userspace
API (which
> >>> results in slightly different actions being taken).
> >>>
> >>>> 3. dom0 provides a virtual PCI device { an interface for
device-OPs
> >>>> and status  of this virtual device} to domU and  through
associated
> >>>> event channel domU makes
> >>>>    "synchronous" use of this device.
> >>>
> >>> domU uses this for control plane operations, but for most work
it can
> >>> talk to its PCI device directly without going through dom0.
> >>>
> >>>> ==> >>>> Queries:
> >>>>
> >>>> 1. What i am really not so sure about is ... passthrough
case
> >>>>     Will there be requirement to map the address space of
this PCI
> >>>> device in domU ?  Will the page which was being shared 
so-far
> >>>> {xen_pci_sharedinfo}
> >>>>      for emulation , be "flipped"  ( transferred
) into domU ?
> >>>
> >>> xen_pci_sharedinfo - is that the page used to talk to the PCI
backend
> >>> from pcifront?  If so, then no, that''s just used for
dom0-domU
> >>> communications.
> >>
> >> Well As I read in the xen0linux code , xen_pci_sharedinfo contains
> >> xen_pci_op where front end specifies which
> >> operation is to be done on which pci device. And then it keeps
checking
> >> the status of this device through another flag in
> >> xen_pci_sharedinfo. I infer here that dom0''s drivers are
being used. Can
> >> you point me towards a code path where domU actually
> >> uses its own drivers having owned the PCI device ?
> >>
> >>> In order to map the address space of the PCI device directly,
the domU
> >>> is given permissions to map the IO memory regions of that
device into
> >>> it''s page tables.  I think this is now possible to do
using a grant
> >>> table operation...
> >>>
> >>> It is also given permission to access certain IO port ranges
so that it
> >>> can use the device''s port IO interfaces.
> >>>
> >>>>      2. Well ,
> >>>> Having read the code for linux (dom0,domU) I see that
there are split
> >>>> device  drivers for PCI. (pci front and PCIback). Which
are normally
> >>>>    communicating over xenbus.  which looks almost like
other split
> >>>> drivers.  How exactly then passthrough enables use of
domU''s drivers ?
> >>>
> >>> The key thing to understand is that the pcifront / pciback is
basically
> >>> just used for setup and teardown, not for the actual IO.  The
real IO
> >>> is done directly by the domU without going through dom0.  For
the block
> >>> and net drivers, *all* IO goes through dom0.
> >>>
> >>>> 3. And if passthrough support isnt provided how will
communication
> >>>> between pcifront-pciback  be different ? ( I guess netbsd
, freebsd do
> >>>> not have passthrough support yet )
> >>>
> >>> I''m not entirely clear what you''re asking
here, but I''ll take a stab at
> >>> it:
> >>>
> >>> if pcifront (in domU) and pciback (in dom0) aren''t
available then
> >>> passthrough won''t work.  The dom0 has to support the
backend functions
> >>> of PCI passthrough and the domU has to know how to talk to it.
It''s
> >>> also implicit that they''re using the same interface
version to talk to
> >>> each other - I''m not sure whether that''s
frozen stable or not.
> >>>
> >>> So *if* NetBSD lacks pciback support, it can''t pass
PCI devices to
> >>> guests that do.  Similarly, *if* it lacks pcifront support, it
can''t
> >>> have devices passed to it.
> >>>
> >>>> 4. What  restricts other domUs from accessing PCI device
given to
> >>>> other domU via passrthrough support.
> >>>
> >>> There are some restrictions on what can be done in PCI config
space to
> >>> prevent a guest fouling things up.  These need to be relaxed
for some
> >>> awkward devices, though.
> >>>
> >>> For the device IO itself, domUs are only allowed to map mmio
regions
> >>> and access io ports that are relevant to their device. 
It''s possible
> >>> for these to overlap with those for other devices, in which
case you''re
> >>> trusting the domU to be well behaved.  More crucially, though,
giving a
> >>> domain a device with DMA capabilities is equivalent to giving
it the
> >>> ability to subvert the entire machine.  DMA can''t be
sandboxed on most
> >>> current hardware, so if you give DMA rights to a VM
it''s automatically
> >>> just as trusted as dom0 with respect to not fooling about with
other
> >>> domains, hardware, etc.
> >>>
> >>> Cheers,
> >>> Mark
> >>
> >> Thanks,
> >> Sanket
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users