Xen devel - Feb 2009 - [RFC] Use device path to assign devices to guest domain

I''m considering making a patch to use device path to assign devices to
guest domain.

We can reserve devices which will be assigned to guest domain by using
devicepath in boot parameter of dom0.
 ex)
  guestdev=PNP0A08:0-0.0,PNP0A08:0-2.0

But we can''t use device path in the guest configuration file now.

To enable it, I will add the following functions.

 - Add "devicepath" and "assignable" field to PPCI class.

 - Use device path, vender ID and device ID to decide uuid value of
   PPCI class.

    Now uuid value of PPCI class is decided by SBDF only. SBDF of
    particular slot can be changed. So uuid value of particular PPCI
    class is changed too if SBDF is changed.
    On the other hand, device path of that isn''t changed. So we can use
    device path to decide whether uuid value of particular PPCI class
    should be changed or not.

    But there is a problem with using device path only. If the device
    of particular slot is exchanged for another device, uuid value
    isn''t changed. In this case, uuid value should be changed. So
    vender ID and device ID are necessary for recognition of exchange
    of device.

 - Enable device path in pci parameter of guest configuration file.
   The format is below.
    pci=[''device path'' or ''SBDF'', ...]

    ex)
     pci=[''PNP0A08:0-2.0-0.0'',
''PNP0A08:0-4.0-0.0'']
      or
     pci=[''0000:00:1.0'', ''0000:00:2.0'']

I hope developers give me some comments.

Thanks,
--
Yuji Shimada


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26 Feb 2009, at 8:40am, Yuji Shimada wrote:
> I hope developers give me some comments.
I don''t really have an opinion about what happens on Linux.

The current Solaris IO domain work uses the BDF derived naming for PCI  
devices in domain specifications. We have been discussing using  
Solaris device paths (such as /pci@0,0/pci8086,3605@2/pci8086,3500@0/ 
pci8086,3518@2/pci108e,4836@0,1) instead, as it would simplify various  
parts of our implementation and administration model.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 26 Feb 2009 10:00:30 +0000
David Edmondson <dme@sun.com> wrote:
> 
> On 26 Feb 2009, at 8:40am, Yuji Shimada wrote:
> > I hope developers give me some comments.
> 
> I don''t really have an opinion about what happens on Linux.
> 
> The current Solaris IO domain work uses the BDF derived naming for PCI  
> devices in domain specifications. We have been discussing using  
> Solaris device paths (such as /pci@0,0/pci8086,3605@2/pci8086,3500@0/ 
> pci8086,3518@2/pci108e,4836@0,1) instead, as it would simplify various  
> parts of our implementation and administration model.
Thank you for your reply.

I''m sorry, I don''t know much about Solaris device path.
My device path doesn''t use bus# but ACPI''s _HID, _UID, device#
and
function#. Therefore, even if a new device is added, device path
remains unchanged. For your reference, I attached our example figure.

My device path is similar to EFI''s one. So my RFC suits fine on x86 and
IA64 architecture.

My RFC keeps current Xen''s SBDF assignment. So you can add Solaris
device path if you need to do.

Thanks,
--
Yuji Shimada


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 27, 2009 at 01:53:32PM +0900, Yuji Shimada
wrote:
> On Thu, 26 Feb 2009 10:00:30 +0000
> David Edmondson <dme@sun.com> wrote:
> 
> > 
> > On 26 Feb 2009, at 8:40am, Yuji Shimada wrote:
> > > I hope developers give me some comments.
> > 
> > I don''t really have an opinion about what happens on Linux.
> > 
> > The current Solaris IO domain work uses the BDF derived naming for PCI
> > devices in domain specifications. We have been discussing using  
> > Solaris device paths (such as /pci@0,0/pci8086,3605@2/pci8086,3500@0/ 
> > pci8086,3518@2/pci108e,4836@0,1) instead, as it would simplify various
> > parts of our implementation and administration model.
Hi David,

is there a situation in Solaris where the BDF can change,
like the situation that Shimada-san has described in Linux?
> Thank you for your reply.
> 
> I''m sorry, I don''t know much about Solaris device path.
> My device path doesn''t use bus# but ACPI''s _HID, _UID,
device# and
> function#. Therefore, even if a new device is added, device path
> remains unchanged. For your reference, I attached our example figure.
> 
> My device path is similar to EFI''s one. So my RFC suits fine on
x86 and
> IA64 architecture.
> 
> My RFC keeps current Xen''s SBDF assignment. So you can add Solaris
> device path if you need to do.
Hi Shimada-san,

I think that your idea sounds very reasonable, especially since it is
backwards-compatible.

Looking at your diagram, where the SBDF of PNP0A08:0-3.0-2.0 changes from
0000:02:02.0 to 0000:03:02.0, I wonder if there are any places where this
change could cause problems in existing xen, ioemu-dm or xend code.

* I wonder what would happen if PNP0A08:0-3.0-2.0 was hot-plugged into a
  domU as 0000:02:02.0 and subsequently an attempt to hot-unplug it from
  the same domU.

* I wonder what would happen if PNP0A08:0-3.0-2.0 was hidden from
  dom0 at boot time by referring to it as 0000:02:02.0 and then
  after its SBDF had changed to 0000:03:02.0 an attempt was made
  to hot-plug or otherwise pass-through the device to a domU.

-- 
Simon Horman
  VA Linux Systems Japan K.K., Sydney, Australia Satellite Office
  H: www.vergenet.net/~horms/             W: www.valinux.co.jp/en


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 5 Mar 2009 16:12:10 +1100
Simon Horman <horms@verge.net.au> wrote:
> I think that your idea sounds very reasonable, especially since it is
> backwards-compatible.
> 
> Looking at your diagram, where the SBDF of PNP0A08:0-3.0-2.0 changes from
> 0000:02:02.0 to 0000:03:02.0, I wonder if there are any places where this
> change could cause problems in existing xen, ioemu-dm or xend code.
> 
> * I wonder what would happen if PNP0A08:0-3.0-2.0 was hot-plugged into a
>   domU as 0000:02:02.0 and subsequently an attempt to hot-unplug it from
>   the same domU.
> 
> * I wonder what would happen if PNP0A08:0-3.0-2.0 was hidden from
>   dom0 at boot time by referring to it as 0000:02:02.0 and then
>   after its SBDF had changed to 0000:03:02.0 an attempt was made
>   to hot-plug or otherwise pass-through the device to a domU.
Hi Simon,

First of all, change of SBDF in my diagram does not show example of
runtime hotplug. If linux is booted in the situation of left side
diagram, hotplug of PCI-PCI bridge[4.0] and device[0.0] fails.
Because, linux does not rebalance bus# at runtime, and there is no
free bus# under PCI-PCI bridge[1.0].
My diagram shows example of inserting bridge and device while machine
is power off.

One of reasons using device path is for "xm new" command and "xm
start"
command.
When guest is prepared by "xm new", xend stores guest information
including device path in a disk. And when the guest starts by "xm
start", stored guest information is used. The guest information remains
even if hypervisor is rebooted. When we use SBDF, the different device
will be assigned if SBDF is changed at boot time.
When we use device path, the same device will be assigned even if SBDF
is changed at boot time.
So, unchanging device path is reasonable.

Thanks,
--
Yuji Shimada


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Mar 05, 2009 at 05:15:18PM +0900, Yuji Shimada
wrote:
> On Thu, 5 Mar 2009 16:12:10 +1100
> Simon Horman <horms@verge.net.au> wrote:
> > I think that your idea sounds very reasonable, especially since it is
> > backwards-compatible.
> > 
> > Looking at your diagram, where the SBDF of PNP0A08:0-3.0-2.0 changes
from
> > 0000:02:02.0 to 0000:03:02.0, I wonder if there are any places where
this
> > change could cause problems in existing xen, ioemu-dm or xend code.
> > 
> > * I wonder what would happen if PNP0A08:0-3.0-2.0 was hot-plugged into
a
> >   domU as 0000:02:02.0 and subsequently an attempt to hot-unplug it
from
> >   the same domU.
> > 
> > * I wonder what would happen if PNP0A08:0-3.0-2.0 was hidden from
> >   dom0 at boot time by referring to it as 0000:02:02.0 and then
> >   after its SBDF had changed to 0000:03:02.0 an attempt was made
> >   to hot-plug or otherwise pass-through the device to a domU.
> 
> Hi Simon,
> 
> First of all, change of SBDF in my diagram does not show example of
> runtime hotplug. If linux is booted in the situation of left side
> diagram, hotplug of PCI-PCI bridge[4.0] and device[0.0] fails.
> Because, linux does not rebalance bus# at runtime, and there is no
> free bus# under PCI-PCI bridge[1.0].
> My diagram shows example of inserting bridge and device while machine
> is power off.
> 
> One of reasons using device path is for "xm new" command and
"xm start"
> command.
> When guest is prepared by "xm new", xend stores guest information
> including device path in a disk. And when the guest starts by "xm
> start", stored guest information is used. The guest information
remains
> even if hypervisor is rebooted. When we use SBDF, the different device
> will be assigned if SBDF is changed at boot time.
> When we use device path, the same device will be assigned even if SBDF
> is changed at boot time.
> So, unchanging device path is reasonable.
Hi Shimada-san,

thanks for the clarification, my problems were assuming
that physical hot-plug was occuring. As you point out,
that is not the case so the issues I suggested can''t occur.

-- 
Simon Horman
  VA Linux Systems Japan K.K., Sydney, Australia Satellite Office
  H: www.vergenet.net/~horms/             W: www.valinux.co.jp/en


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Mar 05, 2009 at 04:12:10PM +1100, Simon Horman
wrote:
> On Fri, Feb 27, 2009 at 01:53:32PM +0900, Yuji Shimada wrote:
> > On Thu, 26 Feb 2009 10:00:30 +0000
> > David Edmondson <dme@sun.com> wrote:
> > 
> > > 
> > > On 26 Feb 2009, at 8:40am, Yuji Shimada wrote:
> > > > I hope developers give me some comments.
> > > 
> > > I don''t really have an opinion about what happens on
Linux.
> > > 
> > > The current Solaris IO domain work uses the BDF derived naming
for PCI
> > > devices in domain specifications. We have been discussing using  
> > > Solaris device paths (such as
/pci@0,0/pci8086,3605@2/pci8086,3500@0/
> > > pci8086,3518@2/pci108e,4836@0,1) instead, as it would simplify
various
> > > parts of our implementation and administration model.
> 
> is there a situation in Solaris where the BDF can change,
> like the situation that Shimada-san has described in Linux?
Across a reboot, yes. Without a reboot, then it looks as though it
might happen in principle, but I''ve not seen it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel