Xen devel - Sep 2005 - [Experimental PATCH] PCI and IO device emulation

Hello!

 The attached patch is a continuation of my previous posts of 
*experimental* patches where I tried to slide a PCI emulation layer 
underneath dom-U. This now has no more changes to the PCI code in Linux, 
but does all the work in Xen. This patch now also adds IO port emulation 
of other ports than those related to PCI. Specifically it does the 
following:

- probes the PCI bus for devices and bridges etc. and makes all those 
devices that are not to be hidden available to domain 0; domain 0''s in
and
out instructions are intercepted and those related to the PCI config 
spaces are ''passed through'' to the real device
- allows to hide PCI devices from domain 0 and makes them available to the 
first user domain that starts up (hide=02:01.0 for example as parameter to 
the grub configuration line where xen.gz is passed); problem is still the 
interrupt for the device that does not reach the user domain
- opens the IO port ranges for the user domain according to the devices 
that it has been given to - I have added my own 64 k bits for port 
enablement for in/out instructions so there''s duplication there right
now
with what has been done in Xen before; 
- places a mixture of fake and emulated PCI devices underneath user 
domains - domain 0 only sees real devices but like said before the 
commands are first intercepted and then passed through
- allows to hook emulation code for each IO port   (if a port is enabled, 
all in''out instruction are passed through, if it is disabled, a hander
is
search and if one is found it is invoked, otherwise a ~0 is returned)
- imports QEMU''s PIC, PIT and RTC - I have fairly big confidence that 
these would be ''functionally ok'', but most certainly not in
terms of
''timing'' (RTC, PIT); the code changes in the related files
were tried to
be kept at a minimum; the ''loose'' ends of PIC, PIT, and RTC
are ''hooked
up'' to deliver interrupts and have the internal QEMU clock advanced 
(see
qemu_pc.c:pic_irq_request() for raising interrupts and xen/arch/x86/time.c 
''qemu_advance_clock()'')
 - imports QEMU''s APIC and IOAPIC - they don''t work for sure
due to some
other missing hooks, but they compile fine

- code that needs to be looked at and improved is generally marked with 
''//!!!'' 
- almost nothing is ''spinlocked''


One should be able to patch Xen with it and try it out. Everything should 
still work as it did before, unless one starts moving PCI devices to user 
domains. Let me know what you think.

Signed-off-by: Stefan Berger <stefanb@us.ibm.com>





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Sep 2005, at 03:50, Stefan Berger wrote:
>  The attached patch is a continuation of my previous posts of
> *experimental* patches where I tried to slide a PCI emulation layer
> underneath dom-U. This now has no more changes to the PCI code in 
> Linux,
> but does all the work in Xen. This patch now also adds IO port 
> emulation
> of other ports than those related to PCI. Specifically it does the
> following:
I don''t think this is the best approach, as it''s a slippery
slope. As
you already note, domain0 doesn''t know about hidden devices so it
can''t
to interrupt routing and setup. Following this scheme, this would be 
extra platform code (potentially requiring a full ACPI interpreter) 
that would have to be added to Xen. That''s a path we''ve
already
considered and decided not to take.

The only mechanism I think we should have in Xen is a protected 
interface for allowing domU''s to access PCI config space. I would make 
it an explicit hypercall interface rather than bothering with emulating 
I/O port accesses -- we have to make modifications to the PCI stack 
anyway (otherwise we get into having to do crap like providing fake 
BIOS tables to provide dummy bus and irq info), and adding a new type 
of pci read/write access method is trivial in Linux. I expect the same 
is true of most other OSes.

This interface will reject all config accesses by default, but domain0 
can change access privileges on a per-device basis. All that Xen has to 
do is then mask off some of the registers for write access (e.g., don;t 
allow domU to arbitrarily rewrite resource base addresses) and possibly 
fake out reads for certain registers (e.g., perhaps the IRQ number 
register).

All other smarts belong in domain0 imo. The only reason for not doing 
the whole lot in domain0 is that a pcifront/pciback split driver would 
be a lot more pain to write and to debug.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote on 09/28/2005 08:41:19 AM:
> 
> On 28 Sep 2005, at 03:50, Stefan Berger wrote:
> 
> >  The attached patch is a continuation of my previous posts of
> > *experimental* patches where I tried to slide a PCI emulation layer
> > underneath dom-U. This now has no more changes to the PCI code in 
> > Linux,
> > but does all the work in Xen. This patch now also adds IO port 
> > emulation
> > of other ports than those related to PCI. Specifically it does the
> > following:
> 
> I don''t think this is the best approach, as it''s a
slippery slope. As
> you already note, domain0 doesn''t know about hidden devices so it
can''t
> to interrupt routing and setup. Following this scheme, this would be 
> extra platform code (potentially requiring a full ACPI interpreter) 
> that would have to be added to Xen. That''s a path we''ve
already
> considered and decided not to take.
Would it help to make the system''s ACPI data available to user domains
(by
copying them into the user domain address space upon domain creation) so 
they can do their own translations of IRQ numbers for those devices they 
see?
> 
> The only mechanism I think we should have in Xen is a protected 
> interface for allowing domU''s to access PCI config space. I would
make
> it an explicit hypercall interface rather than bothering with emulating 
> I/O port accesses -- we have to make modifications to the PCI stack 
> anyway (otherwise we get into having to do crap like providing fake 
> BIOS tables to provide dummy bus and irq info), and adding a new type 
> of pci read/write access method is trivial in Linux. I expect the same 
> is true of most other OSes.
I think the dummy bus could have a potential (double) benefit for running 
unmodified legacy OSes as well and would not just serve ''driver
domains''.
> 
> This interface will reject all config accesses by default, but domain0 
> can change access privileges on a per-device basis. All that Xen has to 
> do is then mask off some of the registers for write access (e.g., don;t 
> allow domU to arbitrarily rewrite resource base addresses) and possibly 
> fake out reads for certain registers (e.g., perhaps the IRQ number 
> register).
At least that part you could also solve by providing a PCI emulation 
layer. Some devices can be accessible, others remain hidden. You can also 
intercept requests to the IRQ number register and maybe return the 
system''s real IRQ number without having it translated in the user
domain.
> 
> All other smarts belong in domain0 imo. The only reason for not doing 
> the whole lot in domain0 is that a pcifront/pciback split driver would 
> be a lot more pain to write and to debug.
:-) I would push those smarts downwards to avoid changes in OS(es).

  Stefan
> 
>   -- Keir
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Stefan Berger
> Sent: 30 September 2005 02:16
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [Experimental PATCH] PCI and IO device
emulation
> 
> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote on 09/28/2005 08:41:19
AM:
> 
> >
> > On 28 Sep 2005, at 03:50, Stefan Berger wrote:
> >
> > >  The attached patch is a continuation of my previous posts of
> > > *experimental* patches where I tried to slide a PCI emulation
layer
> > > underneath dom-U. This now has no more changes to the PCI code in
> > > Linux,
> > > but does all the work in Xen. This patch now also adds IO port
> > > emulation
> > > of other ports than those related to PCI. Specifically it does
the
> > > following:
> >
> > I don''t think this is the best approach, as it''s a
slippery slope.
As
> > you already note, domain0 doesn''t know about hidden devices
so it
can''t
> > to interrupt routing and setup. Following this scheme, this would be
> > extra platform code (potentially requiring a full ACPI interpreter)
> > that would have to be added to Xen. That''s a path
we''ve already
> > considered and decided not to take.
> 
> Would it help to make the system''s ACPI data available to user
domains
(by
> copying them into the user domain address space upon domain creation)
so
> they can do their own translations of IRQ numbers for those devices
they
> see?
Will they come up with the same result as dom0 which sets up the IOAPICs
etc?
> 
> >
> > The only mechanism I think we should have in Xen is a protected
> > interface for allowing domU''s to access PCI config space. I
would
make
> > it an explicit hypercall interface rather than bothering with
emulating
> > I/O port accesses -- we have to make modifications to the PCI stack
> > anyway (otherwise we get into having to do crap like providing fake
> > BIOS tables to provide dummy bus and irq info), and adding a new
type
> > of pci read/write access method is trivial in Linux. I expect the
same
> > is true of most other OSes.
> 
> I think the dummy bus could have a potential (double) benefit for
running
> unmodified legacy OSes as well and would not just serve ''driver
domains''.

At the moment the PCI emulation for unmodified guests is dealt with in
the device model in dom0. Are you suggesting to pull this into Xen
instead?

> >
> > This interface will reject all config accesses by default, but
domain0
> > can change access privileges on a per-device basis. All that Xen has
to
> 
> 
> > do is then mask off some of the registers for write access (e.g.,
don;t
> > allow domU to arbitrarily rewrite resource base addresses) and
possibly
> > fake out reads for certain registers (e.g., perhaps the IRQ number
> > register).
> 
> At least that part you could also solve by providing a PCI emulation
> layer. Some devices can be accessible, others remain hidden.
Sure, but then you have to PCI emulation layer in Xen. Just doing what
keir suggests requires one file...
> You can also
> intercept requests to the IRQ number register and maybe return the
> system''s real IRQ number without having it translated in the user
domain.

That''s what Keir suggested. The problem is that at least Linux seems to
largely ignore the IRQ number in the PCI config space.
> >
> > All other smarts belong in domain0 imo. The only reason for not
doing
> > the whole lot in domain0 is that a pcifront/pciback split driver
would
> > be a lot more pain to write and to debug.
> 
> :-) I would push those smarts downwards to avoid changes in OS(es).
With Keir'' proposal there are three types of changes:

- changes required to non-dom0 OSes. I think these are minimal in
keir''s
proposal and only require changes to arch-dependent code. Linux already
supports three PCI config probing methods on ia32. adding a 4th one
which uses a hypercall interface is trivial. The other change would be
to the IRQ setup. You''d have to convince the OS to believe the IRQ
number in the config space (or get it by some other means). I think that
is straight forward as well and both has been done in Xen 2.X. You can
even delete/remove an lot of code related to quirks in the real HW.

- changes to dom0 kernels. These are trickier in Xen 3.x. dom0 has to do
the entire setup for the device as it normally would but then not start
a device driver for devices which are supposed to be hidden from it.
This is likely to require a small change to arch-independent code in
Linux. It would also be nice if dom0 could still see all devices in the
system via things like lspci to make it easier to do device assignment

Another requirement is that both these changes need to coexist in the
same kernel so that no separate kernels are required for dom0 and domUs

- addition to the xen hypercall interface to allow domUs to probe the
PCI config space. This is not strictly required as one could just
emulate the io reads/writes of a guests, but it seems cleaner to do it
via a hypercall interface. There is also only a small addition to the
hypervisor to deal with the hypercalls. But in xen 2x this is about 

Your proposal doesn''t require changes to guests but adds significantly
more code to Xen. I think the changes required to device driver domains
are small.

Could you elaborate a bit more on how your proposal interacts with Dom0
and the IRQ routing/IO APIC set up it is doing at the moment. If you
hide the device completely, as I understand you are doing, how will that
be done?

Thanks
Rolf


> 
>   Stefan
> >
> >   -- Keir
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Neugebauer, Rolf" <rolf.neugebauer@intel.com> wrote on
09/30/2005
05:53:36 AM:
> 
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > bounces@lists.xensource.com] On Behalf Of Stefan Berger
> > Sent: 30 September 2005 02:16
> > To: Keir Fraser
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] [Experimental PATCH] PCI and IO device
> emulation
> > 
> > Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote on 09/28/2005
08:41:19
> AM:
> > 
> > >
> > > On 28 Sep 2005, at 03:50, Stefan Berger wrote:
> > >
> 
> > 
> > >
> > > The only mechanism I think we should have in Xen is a protected
> > > interface for allowing domU''s to access PCI config
space. I would
> make
> > > it an explicit hypercall interface rather than bothering with
> emulating
> > > I/O port accesses -- we have to make modifications to the PCI
stack
> > > anyway (otherwise we get into having to do crap like providing
fake
> > > BIOS tables to provide dummy bus and irq info), and adding a new
> type
> > > of pci read/write access method is trivial in Linux. I expect the
> same
> > > is true of most other OSes.
> > 
> > I think the dummy bus could have a potential (double) benefit for
> running
> > unmodified legacy OSes as well and would not just serve
''driver
> domains''.
> 
> At the moment the PCI emulation for unmodified guests is dealt with in
> the device model in dom0. Are you suggesting to pull this into Xen
> instead?
If we can come up with architecture that can serve both purposes, enable 
driver domains and provide access to an emulated PCI bus (or even real 
devices) for unmodified guests, then I would do it.
> 
> 
> > >
> > > This interface will reject all config accesses by default, but
> domain0
> > > can change access privileges on a per-device basis. All that Xen
has
> to
> > 
> > 
> > > do is then mask off some of the registers for write access (e.g.,
> don;t
> > > allow domU to arbitrarily rewrite resource base addresses) and
> possibly
> > > fake out reads for certain registers (e.g., perhaps the IRQ
number
> > > register).
> > 
> > At least that part you could also solve by providing a PCI emulation
> > layer. Some devices can be accessible, others remain hidden.
> 
> Sure, but then you have to PCI emulation layer in Xen. Just doing what
> keir suggests requires one file...
> 
> > You can also
> > intercept requests to the IRQ number register and maybe return the
> > system''s real IRQ number without having it translated in the
user
> domain.
> 
> That''s what Keir suggested. The problem is that at least Linux
seems to
> largely ignore the IRQ number in the PCI config space.
I am saying that this can also be done in the PCI emulation layer.
> 
> > >
> > > All other smarts belong in domain0 imo. The only reason for not
> doing
> > > the whole lot in domain0 is that a pcifront/pciback split driver
> would
> > > be a lot more pain to write and to debug.
> > 
> > :-) I would push those smarts downwards to avoid changes in OS(es).
> 
> With Keir'' proposal there are three types of changes:
> 
> - changes required to non-dom0 OSes. I think these are minimal in
keir''s
> proposal and only require changes to arch-dependent code. Linux already
> supports three PCI config probing methods on ia32. adding a 4th one
> which uses a hypercall interface is trivial. The other change would be
> to the IRQ setup. You''d have to convince the OS to believe the IRQ
> number in the config space (or get it by some other means). I think that
I also think it would be important that the OS use the IRQ number in the 
config space. The Hypervisor can always do translation of IRQ numbers from 
machine physical to VM in the hypervisor before propagating the IRQ to the 
domain.
> is straight forward as well and both has been done in Xen 2.X. You can
> even delete/remove an lot of code related to quirks in the real HW.
>
> - changes to dom0 kernels. These are trickier in Xen 3.x. dom0 has to do
> the entire setup for the device as it normally would but then not start
> a device driver for devices which are supposed to be hidden from it.
> This is likely to require a small change to arch-independent code in
> Linux. It would also be nice if dom0 could still see all devices in the
> system via things like lspci to make it easier to do device assignment
> 
> Another requirement is that both these changes need to coexist in the
> same kernel so that no separate kernels are required for dom0 and domUs
> 
> - addition to the xen hypercall interface to allow domUs to probe the
> PCI config space. This is not strictly required as one could just
> emulate the io reads/writes of a guests, but it seems cleaner to do it
> via a hypercall interface. There is also only a small addition to the
> hypervisor to deal with the hypercalls. But in xen 2x this is about 
Access to PCI config space is protected by spinlocks. First one writes 
bus/device/function and address into the config register then one reads or 
writes from/to the data register affecting the previously given address. 
To properly protect access to the config space it really should be done in 
a common place and accessed in the same way by domain 0 and any other 
domain.
> 
> Your proposal doesn''t require changes to guests but adds
significantly
> more code to Xen. I think the changes required to device driver domains
> are small.
> 
> Could you elaborate a bit more on how your proposal interacts with Dom0
> and the IRQ routing/IO APIC set up it is doing at the moment. If you
> hide the device completely, as I understand you are doing, how will that
> be done?
I don''t think you would like to move ownership of the IO-APIC to Xen
and
let Xen do that part of the setup.  Therefore, I would build a request 
list for domain 0 to do the necessary setup for those devices it does not 
see and provide the HV with the necessary information (like machine 
physical interrupt) it needs to properly route the IRQs to a domain that 
is accessing a device. This would require an additional module in the 
domain 0 kernel and some more hypercalls.

   Stefan
 
> 
> Thanks
> Rolf
> 
> 
> 
> > 
> >   Stefan
> > >
> > >   -- Keir
> > >
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30 Sep 2005, at 18:14, Stefan Berger wrote:
> Access to PCI config space is protected by spinlocks. First one writes 
> bus/device/function and address into the config register then one 
> reads or writes from/to the data register affecting the previously 
> given address. To properly protect access to the config space it 
> really should be done in a common place and accessed in the same way 
> by domain 0 and any other domain.
That''s orthogonal to whether we emulate or have an explicit hypercall. 
Yes, dom0 would have to use the same interface as the domUs.
> I don''t think you would like to move ownership of the IO-APIC to
Xen
> and let Xen do that part of the setup.  Therefore, I would build a 
> request list for domain 0 to do the necessary setup for those devices 
> it does not see and provide the HV with the necessary information 
> (like machine physical interrupt) it needs to properly route the IRQs 
> to a domain that is accessing a device. This would require an 
> additional module in the domain 0 kernel and some more hypercalls.
Seems simpler just to get dom0 to do the final setup when it builds the 
driver domain.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel