Xen devel - Feb 2005 - RFC: Creation of virtual bus, hook-up of Xen devices

For many purposes on a Linux system, it is required to have devices
export themselves via the device infrastructure (exposed via sysfs) to
allow for reasonable user-space probing and discovery of available
devices.  This is especially useful/necessary for things like installing
to guest systems.

The attached patch is a first pass at starting to get something along
these lines for Xen devices to hook into.  Provided sysfs bits
are /sys/bus/x/devices/netN for each xen_net device
and /sys/bus/x/drivers/xen_{net,blk} as points for the xen_net and
xen_blk drivers.  I haven''t actually really hooked up the xen_blk
devices yet as I think that getting some of the base infrastructure
committed first will make that a little better.  Also, there''s
definitely more that can be done in terms of cleaning up netfront to
better take advantage of the fact that there''s a struct device
associated with each net_device instead of keeping track of all of the
generic bits in the net_private struct.

Comments? 

Jeremy

On Tue, 01 Feb 2005 15:30:51 -0500, Jeremy Katz <katzj@redhat.com>
wrote:
> For many purposes on a Linux system, it is required to have devices
> export themselves via the device infrastructure (exposed via sysfs) to
> allow for reasonable user-space probing and discovery of available
> devices.  This is especially useful/necessary for things like installing
> to guest systems.
Can you give more specific examples of tools which would use this? 
What information would be exported under devices/netN and
devices/blkN?

Having support for the device infrastructure is good - thanks for the patch.
> Comments?
There''s a few issues with the patch:
- netfront.c is shared between linux 2.6 and 2.4 and it''s not going to
build on 2.4 with the changes.  same for blkfront.c.
- wouldn''t /sys/bus/xen be a better name than /sys/bus/x?
- same for exported functions, x_register_driver et al. are IMHO too
general and pollute the namespace -- I''d suggest prefixing everything
with xenbus, incl the structs which get defined in xbus.h (xenbus.h).

- if I understand the change to the network probing code correctly, it
relies on xbus_init getting called before netif_init.  xbus_init will
change the driver status up and then down again, causing the network
devices to get enumerated and register the network devices during this
enumeration.  I assume that the call to device_register from
x_register_device then causes the poll hook to get called which
creates the device struct.  netif_init will then change the driver
status up again and cause another enumeration which then puts the vif
into connected state.
You''ve already noted that the network probing code doesn''t
belong in
the xbus code.  We also don''t want the additional driver status
changes.  Wouldn''t it just work to call xenbus_register_device in the
interface status message handler?

- have you tested suspend/resume?

I think we need to address these issues before we proceed.

     christian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, 2005-02-01 at 22:49 +0000, Christian Limpach
wrote:
> On Tue, 01 Feb 2005 15:30:51 -0500, Jeremy Katz <katzj@redhat.com>
wrote:
> > For many purposes on a Linux system, it is required to have devices
> > export themselves via the device infrastructure (exposed via sysfs) to
> > allow for reasonable user-space probing and discovery of available
> > devices.  This is especially useful/necessary for things like
installing
> > to guest systems.
> 
> Can you give more specific examples of tools which would use this? 
> What information would be exported under devices/netN and
> devices/blkN?
kudzu (and thus anaconda) and hal both immediately spring to mind.
anaconda being the one that''s near and dear to me ;)

I want to get to where I can install inside a guest and have it be
virtually identical to installation on hardware.  Different drivers is
fine but probing is still useful.
> Having support for the device infrastructure is good - thanks for the
patch.
> 
> > Comments?
> 
> There''s a few issues with the patch:
> - netfront.c is shared between linux 2.6 and 2.4 and it''s not
going to
> build on 2.4 with the changes.  same for blkfront.c.
I thought that was the case.  Like I said, this is really a first pass
just to get something to look at since I think it''s far more concrete
and less hand-wavey with some code to look at.  Although to really fully
take advantage of some of the stuff in 2.6, I''m not entirely sure how
to
keep this in sync.
> - wouldn''t /sys/bus/xen be a better name than /sys/bus/x?
> - same for exported functions, x_register_driver et al. are IMHO too
> general and pollute the namespace -- I''d suggest prefixing
everything
> with xenbus, incl the structs which get defined in xbus.h (xenbus.h).
s/x/xen/ is easy enough to do.  Although not as xen leaves it more open
for other virtualization frameworks to eventually use the same model.
The idea of sub_x_bus also came up just from the nice anagram it would
give :-)
> - if I understand the change to the network probing code correctly, it
> relies on xbus_init getting called before netif_init.  xbus_init will
> change the driver status up and then down again, causing the network
> devices to get enumerated and register the network devices during this
> enumeration.  I assume that the call to device_register from
> x_register_device then causes the poll hook to get called which
> creates the device struct.  netif_init will then change the driver
> status up again and cause another enumeration which then puts the vif
> into connected state.
Exactly.
> You''ve already noted that the network probing code
doesn''t belong in
> the xbus code.  We also don''t want the additional driver status
> changes.  Wouldn''t it just work to call xenbus_register_device in
the
> interface status message handler?
The problem is that you really want to be able to probe what devices are
there without loading the module.  Otherwise, if you build the net
device as a module, how do you know that there are net devices present?
You can do load/unload but that''s a little bit ugly (and racy since it
involves removing modules).  

Another thought would be to add a way to get a simple enumeration of
devices with types.  That would then be able to be generic and used to
register all the devices.  Doing that starts to involve some bigger
interface changes, though, which I wanted to avoid at first.
> - have you tested suspend/resume?
Nope, the last time I tried suspend/resume with the copy of -unstable I
was running against it wasn''t working at all.  I need to pull up to
current and then that should be easy enough to get in place.
> I think we need to address these issues before we proceed.
No question that things have to be done... but if there''s not screaming
at the basic approach, it''s far easier for me to clean stuff afterwards
than to clean stuff up and then have to rework it majorly.  

Jeremy



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> The problem is that you really want to be able to probe what devices are
> there without loading the module.  Otherwise, if you build the net
> device as a module, how do you know that there are net devices present?
> You can do load/unload but that''s a little bit ugly (and racy
since it
> involves removing modules).  
Yep, we certainly don''t want to get into module
loading/unloading. It''s probably reasonable to always load the
net/blkfront drivers when running on xen -- its perhaps best just
to build them into the kernel.
 
> Another thought would be to add a way to get a simple enumeration of
> devices with types.  That would then be able to be generic and used to
> register all the devices.  Doing that starts to involve some bigger
> interface changes, though, which I wanted to avoid at first.
We''re already planing changes to the control messages that will
break compatibility with 2.0 tools, so its reasonable to discuss
further changes.
> > - have you tested suspend/resume?
> 
> Nope, the last time I tried suspend/resume with the copy of -unstable I
> was running against it wasn''t working at all.  I need to pull up
to
> current and then that should be easy enough to get in place.
I''m afraid its still broken in unstable as no-one has tided up
after the SMP guest changes. We need to extend the dom0_op to
enable the context of different VCPUs to be read out, and modify
the save restore code accordingly. We also need to ensure that
the suspend notifier does an smp_call_function to park
the other VCPUs for the suspend.
 
> > I think we need to address these issues before we proceed.
> 
> No question that things have to be done... but if there''s not
screaming
> at the basic approach, it''s far easier for me to clean stuff
afterwards
> than to clean stuff up and then have to rework it majorly.  
Yep, discussing the design in advance and sending patches early
is how we like to do things round here -- thanks!

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, Feb 01, 2005 at 06:09:20PM -0500, Jeremy Katz
wrote:
> > Can you give more specific examples of tools which would use this? 
> 
> kudzu (and thus anaconda) and hal both immediately spring to mind.
> anaconda being the one that''s near and dear to me ;)
> 
> I want to get to where I can install inside a guest and have it be
> virtually identical to installation on hardware.  Different drivers is
> fine but probing is still useful.
Is this where anaconda tells you it can''t find a network device even
if it''s built into the kernel?
> > What information would be exported under devices/netN and
> > devices/blkN?
Do you know yet what information would be exported?  Or is the existence
of the netN/blkN devices sufficient for your immediate usage?
> > - netfront.c is shared between linux 2.6 and 2.4 and it''s not
going to
> > build on 2.4 with the changes.  same for blkfront.c.
> 
> I thought that was the case.  Like I said, this is really a first pass
> just to get something to look at since I think it''s far more
concrete
> and less hand-wavey with some code to look at.  Although to really fully
> take advantage of some of the stuff in 2.6, I''m not entirely sure
how to
> keep this in sync.
So far we''ve used macros to either make the 2.6 calls do something
useful in 2.4 or do nothing at all.
> > - wouldn''t /sys/bus/xen be a better name than /sys/bus/x?
> > - same for exported functions, x_register_driver et al. are IMHO too
> > general and pollute the namespace -- I''d suggest prefixing
everything
> > with xenbus, incl the structs which get defined in xbus.h (xenbus.h).
> 
> s/x/xen/ is easy enough to do.  Although not as xen leaves it more open
> for other virtualization frameworks to eventually use the same model.
> The idea of sub_x_bus also came up just from the nice anagram it would
> give :-)
I see.  x is just so very unspecific.  We''re certainly happy with xen
;-)
Maybe vmm would make sense?
> > You''ve already noted that the network probing code
doesn''t belong in
> > the xbus code.  We also don''t want the additional driver
status
> > changes.  Wouldn''t it just work to call
xenbus_register_device in the
> > interface status message handler?
> 
> The problem is that you really want to be able to probe what devices are
> there without loading the module.  Otherwise, if you build the net
> device as a module, how do you know that there are net devices present?
> You can do load/unload but that''s a little bit ugly (and racy
since it
> involves removing modules).  
Ah I see.  How about you move the network probing code into a seperate file
next to netfront.c and always include that in the kernel?  This would also
take care of some of the 2.4 code sharing issues since 2.4 could then have
a different file.  Not sure yet how best to avoid doing enumeration twice,
it looks like we want an additional driver state or some probe only
enumeration call.

I reckon my description wasn''t 100% accurate after all -- I assumed
that
the probe hook was getting called as soon as the device was registered.
That wouldn''t make sense if the module is not loaded yet (and the probe
function isn''t known to the bus).  When does the probe function get
called?
> Another thought would be to add a way to get a simple enumeration of
> devices with types.  That would then be able to be generic and used to
> register all the devices.  Doing that starts to involve some bigger
> interface changes, though, which I wanted to avoid at first.
How do other devices handle this?  Isn''t there a generic framework for
this?  Or are you suggesting adding a non-xen specific framework?
> > - have you tested suspend/resume?
> 
> Nope, the last time I tried suspend/resume with the copy of -unstable I
> was running against it wasn''t working at all.  I need to pull up
to
> current and then that should be easy enough to get in place.
Yes, suspend/resume is broken in -unstable.
> > I think we need to address these issues before we proceed.
> 
> No question that things have to be done... but if there''s not
screaming
> at the basic approach, it''s far easier for me to clean stuff
afterwards
> than to clean stuff up and then have to rework it majorly.  
I think that the way how you hook device registration into the code
which discovers devices (or vice versa) is the tricky part and needs
to be decided upon early.

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, 2005-02-01 at 23:53 +0000, Ian Pratt wrote:
>  > The problem is that you really want to be able to probe what devices
are
> > there without loading the module.  Otherwise, if you build the net
> > device as a module, how do you know that there are net devices
present?
> > You can do load/unload but that''s a little bit ugly (and racy
since it
> > involves removing modules).  
> 
> Yep, we certainly don''t want to get into module
> loading/unloading. It''s probably reasonable to always load the
> net/blkfront drivers when running on xen -- its perhaps best just
> to build them into the kernel.
Well, you want modules just so that things "look like" everywhere
else.
The route of built-in with one kernel and modular on another leads to
madness eventually.  That doesn''t necessarily mean they need to be
unloadable (it''s easy enough to permanently refcount a module, and
it''s
not unheard of), but if they''re not, you probably do want to be able to
tell what''s there
 
> > Another thought would be to add a way to get a simple enumeration of
> > devices with types.  That would then be able to be generic and used to
> > register all the devices.  Doing that starts to involve some bigger
> > interface changes, though, which I wanted to avoid at first.
> 
> We''re already planing changes to the control messages that will
> break compatibility with 2.0 tools, so its reasonable to discuss
> further changes.
That was the impression I got.  Really, adding a generic "added a net
device, it''s handle is x", "added a block device,
it''s handle is y", ...
etc[1] is probably the easiest way to do this.  It also then interacts
nicely for hotplug as you have the bus do all the matching and then it
just hands off the struct device to the appropriate driver''s ->probe
function.  Which then lets you have the tools in dom0 support more
devices than you have backend drivers in the guest for without ill
effect (and you could even still have the devices in your device tree so
that if you then built a module in domU, it could claim those
devices).  

Jeremy

[1] Okay, doing a numeric type instead of a string is slightly prettier,
but for the sake of argument



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-02-02 at 01:06 +0000, Christian Limpach
wrote:
> On Tue, Feb 01, 2005 at 06:09:20PM -0500, Jeremy Katz wrote:
> > > Can you give more specific examples of tools which would use
this?
> > 
> > kudzu (and thus anaconda) and hal both immediately spring to mind.
> > anaconda being the one that''s near and dear to me ;)
> > 
> > I want to get to where I can install inside a guest and have it be
> > virtually identical to installation on hardware.  Different drivers is
> > fine but probing is still useful.
> 
> Is this where anaconda tells you it can''t find a network device
even
> if it''s built into the kernel?
Related.  The requirement of it being modular is largely due to
silliness with not being able to tell if your PCI network card is bound
to a driver or not.  And actually, that should be able to be made a
little bit better (and thus, support proper drivers which are built-in
for odd cases)
> > > What information would be exported under devices/netN and
> > > devices/blkN?
> 
> Do you know yet what information would be exported?  Or is the existence
> of the netN/blkN devices sufficient for your immediate usage?
Existence is good enough for _my_ immediate needs, but it''s probably
not
the best in general or for the longer term.  If things can get to where
there''s a unique identifier for every virtual device (which is probably
the better long term option), then a cleaner arrangement would probably
be
 /sys/bus/xen/devices
   devid1/
      class (file containing a class id for net, block, usb, ...)
      handle
      evtchn
      irq
      driver -> driver dir if driver is loaded

Then it can parallel other actual hardware buses a little bit more
closely.
> > > - netfront.c is shared between linux 2.6 and 2.4 and
it''s not going to
> > > build on 2.4 with the changes.  same for blkfront.c.
> > 
> > I thought that was the case.  Like I said, this is really a first pass
> > just to get something to look at since I think it''s far more
concrete
> > and less hand-wavey with some code to look at.  Although to really
fully
> > take advantage of some of the stuff in 2.6, I''m not entirely
sure how to
> > keep this in sync.
> 
> So far we''ve used macros to either make the 2.6 calls do something
> useful in 2.4 or do nothing at all.
Okay, I''ll have to look a little closer for that, then. 
Tomorrow''s todo
list begins :)
> > > - wouldn''t /sys/bus/xen be a better name than
/sys/bus/x?
> > > - same for exported functions, x_register_driver et al. are IMHO
too
> > > general and pollute the namespace -- I''d suggest
prefixing everything
> > > with xenbus, incl the structs which get defined in xbus.h
(xenbus.h).
> > 
> > s/x/xen/ is easy enough to do.  Although not as xen leaves it more
open
> > for other virtualization frameworks to eventually use the same model.
> > The idea of sub_x_bus also came up just from the nice anagram it would
> > give :-)
> 
> I see.  x is just so very unspecific.  We''re certainly happy with
xen ;-)
> Maybe vmm would make sense?
Or maybe vmmio to be a little bit more descriptive.  Since I don''t
think
that things like memory, cpus, etc make sense on there and so it''s
mostly for IO devices.
> > > You''ve already noted that the network probing code
doesn''t belong in
> > > the xbus code.  We also don''t want the additional driver
status
> > > changes.  Wouldn''t it just work to call
xenbus_register_device in the
> > > interface status message handler?
> > 
> > The problem is that you really want to be able to probe what devices
are
> > there without loading the module.  Otherwise, if you build the net
> > device as a module, how do you know that there are net devices
present?
> > You can do load/unload but that''s a little bit ugly (and racy
since it
> > involves removing modules).  
> 
> Ah I see.  How about you move the network probing code into a seperate file
> next to netfront.c and always include that in the kernel?  This would also
> take care of some of the 2.4 code sharing issues since 2.4 could then have
> a different file.  Not sure yet how best to avoid doing enumeration twice,
> it looks like we want an additional driver state or some probe only
> enumeration call.
That could work, although if the end direction is something akin to the
above, then it may make less sense to do this as an intermediate step.
If not, then it shouldn''t be too hard to do. 
> I reckon my description wasn''t 100% accurate after all -- I
assumed that
> the probe hook was getting called as soon as the device was registered.
> That wouldn''t make sense if the module is not loaded yet (and the
probe
> function isn''t known to the bus).  When does the probe function
get
> called?
When the device gets registered, it gets added as a child of the bus
device.  device_register() uses bus->match to see if any of the loaded &
registered drivers can drive the device.  Since no drivers are loaded
when x_bus_init() gets called, the driver loads and registers later.  On
driver registration, the bus gets walked for unattached devices that
match the type the driver supports.  On a match, driver->probe is called
and the device and driver are bound.   
> > Another thought would be to add a way to get a simple enumeration of
> > devices with types.  That would then be able to be generic and used to
> > register all the devices.  Doing that starts to involve some bigger
> > interface changes, though, which I wanted to avoid at first.
> 
> How do other devices handle this?  Isn''t there a generic framework
for
> this?  Or are you suggesting adding a non-xen specific framework?
It tends to be bus-specific as dictated by the hardware.  eg, for USB,
you get control messages that the host controller interprets and then
passes off as appropriate.  For PCI, you can walk a tree provided in a
host-specific way (BIOS/PIRQ tables or ACPI on x86, OpenFirmware on PPC,
etc).  So there would need to be a xen specific way of doing this or
making xen present something that looks like one of these other things.
I think that the approach that most mimics what the rest of the Xen
approach is would be to add a Xen specific way, though.
> > > I think we need to address these issues before we proceed.
> > 
> > No question that things have to be done... but if there''s not
screaming
> > at the basic approach, it''s far easier for me to clean stuff
afterwards
> > than to clean stuff up and then have to rework it majorly.  
> 
> I think that the way how you hook device registration into the code
> which discovers devices (or vice versa) is the tricky part and needs
> to be decided upon early.
Agreed.  

Jeremy



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Tue, Feb 01, 2005 at 09:20:20PM -0500, Jeremy Katz
wrote:
> > > > What information would be exported under devices/netN and
> > > > devices/blkN?
> > 
> > Do you know yet what information would be exported?  Or is the
existence
> > of the netN/blkN devices sufficient for your immediate usage?
> 
> Existence is good enough for _my_ immediate needs, but it''s
probably not
> the best in general or for the longer term.  If things can get to where
> there''s a unique identifier for every virtual device (which is
probably
> the better long term option), then a cleaner arrangement would probably
> be
>  /sys/bus/xen/devices
>    devid1/
>       class (file containing a class id for net, block, usb, ...)
>       handle
>       evtchn
>       irq
>       driver -> driver dir if driver is loaded
> 
> Then it can parallel other actual hardware buses a little bit more
> closely.
OK, sounds good.
> > > > - wouldn''t /sys/bus/xen be a better name than
/sys/bus/x?
> > > > - same for exported functions, x_register_driver et al. are
IMHO too
> > > > general and pollute the namespace -- I''d suggest
prefixing everything
> > > > with xenbus, incl the structs which get defined in xbus.h
(xenbus.h).
> > > 
> > > s/x/xen/ is easy enough to do.  Although not as xen leaves it
more open
> > > for other virtualization frameworks to eventually use the same
model.
> > > The idea of sub_x_bus also came up just from the nice anagram it
would
> > > give :-)
> > 
> > I see.  x is just so very unspecific.  We''re certainly happy
with xen ;-)
> > Maybe vmm would make sense?
> 
> Or maybe vmmio to be a little bit more descriptive.  Since I don''t
think
> that things like memory, cpus, etc make sense on there and so it''s
> mostly for IO devices.
Actually, we might want to add memory and cpus, or is there another
abstraction to represent hotplug cpus and memory?  I think xen or vmm
would be good names, I''ve used hypervisor in NetBSD.
> > > > You''ve already noted that the network probing code
doesn''t belong in
> > > > the xbus code.  We also don''t want the additional
driver status
> > > > changes.  Wouldn''t it just work to call
xenbus_register_device in the
> > > > interface status message handler?
> > > 
> > > The problem is that you really want to be able to probe what
devices are
> > > there without loading the module.  Otherwise, if you build the
net
> > > device as a module, how do you know that there are net devices
present?
> > > You can do load/unload but that''s a little bit ugly (and
racy since it
> > > involves removing modules).  
> > 
> > Ah I see.  How about you move the network probing code into a seperate
file
> > next to netfront.c and always include that in the kernel?  This would
also
> > take care of some of the 2.4 code sharing issues since 2.4 could then
have
> > a different file.  Not sure yet how best to avoid doing enumeration
twice,
> > it looks like we want an additional driver state or some probe only
> > enumeration call.
> 
> That could work, although if the end direction is something akin to the
> above, then it may make less sense to do this as an intermediate step.
> If not, then it shouldn''t be too hard to do. 
I''m not 100% sure what you mean by end direction?  Do you mean the
change
to a probe only enumeration call?  I think that would be a simple change
since the call would be very similar on the client side, but it would
allow the backend to skip any changes which are only required when the
device is actually brought up.
> > > Another thought would be to add a way to get a simple enumeration
of
> > > devices with types.  That would then be able to be generic and
used to
> > > register all the devices.  Doing that starts to involve some
bigger
> > > interface changes, though, which I wanted to avoid at first.
> > 
> > How do other devices handle this?  Isn''t there a generic
framework for
> > this?  Or are you suggesting adding a non-xen specific framework?
> 
> It tends to be bus-specific as dictated by the hardware.  eg, for USB,
> you get control messages that the host controller interprets and then
> passes off as appropriate.  For PCI, you can walk a tree provided in a
> host-specific way (BIOS/PIRQ tables or ACPI on x86, OpenFirmware on PPC,
> etc).  So there would need to be a xen specific way of doing this or
> making xen present something that looks like one of these other things.
> I think that the approach that most mimics what the rest of the Xen
> approach is would be to add a Xen specific way, though.
I think you want to query each backend and find out what devices it has.
I''m not sure if the enumeration function should be generic or
device-specific.  Our current device-specific approach could probably be
changed to be generic but in a first instance it would be quickest to
implement this using the existing "queries" since there are only two
device types.

Have you made any progress on this?

    christian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Thu, 2005-02-10 at 00:17 +0000, Christian Limpach
wrote:
>On Tue, Feb 01, 2005 at 09:20:20PM -0500, Jeremy Katz wrote:
>> > > > What information would be exported under devices/netN
and
>> > > > devices/blkN?
>> > 
>> > Do you know yet what information would be exported?  Or is the
existence
>> > of the netN/blkN devices sufficient for your immediate usage?
>> 
>> Existence is good enough for _my_ immediate needs, but it''s
probably not
>> the best in general or for the longer term.  If things can get to where
>> there''s a unique identifier for every virtual device (which is
probably
>> the better long term option), then a cleaner arrangement would probably
>> be
>>  /sys/bus/xen/devices
>>    devid1/
>>       class (file containing a class id for net, block, usb, ...)
>>       handle
>>       evtchn
>>       irq
>>       driver -> driver dir if driver is loaded
>> 
>> Then it can parallel other actual hardware buses a little bit more
>> closely.
>
>OK, sounds good.
After getting some work done, one problem is there''s not a
device-type-independent device id for each device (or if so, I''m
missing
it entirely :).  I can get type + index to get an identifier, but
that''s
not generic across all the devices.  That''s not a huge stumbling block,
though, as it''s at least _a_ unique identifier.  

I''m also not sure how things interact as far as device indexing after
suspend of a guest since that''s not working in -unstable.
>> > > > - wouldn''t /sys/bus/xen be a better name than
/sys/bus/x?
>> > > > - same for exported functions, x_register_driver et al.
are IMHO too
>> > > > general and pollute the namespace -- I''d
suggest prefixing everything
>> > > > with xenbus, incl the structs which get defined in
xbus.h (xenbus.h).
>> > > 
>> > > s/x/xen/ is easy enough to do.  Although not as xen leaves it
more open
>> > > for other virtualization frameworks to eventually use the
same model.
>> > > The idea of sub_x_bus also came up just from the nice anagram
it would
>> > > give :-)
>> > 
>> > I see.  x is just so very unspecific.  We''re certainly
happy with xen ;-)
>> > Maybe vmm would make sense?
>> 
>> Or maybe vmmio to be a little bit more descriptive.  Since I
don''t think
>> that things like memory, cpus, etc make sense on there and so
it''s
>> mostly for IO devices.
>
>Actually, we might want to add memory and cpus, or is there another
>abstraction to represent hotplug cpus and memory?  I think xen or vmm
>would be good names, I''ve used hypervisor in NetBSD.
I think the general handling of hotplug cpus and memory in Linux right
now falls into the category of "poor".  :-)   I''ll go with
vmm, though,
to keep it generic enough for exposing cpu and memory via it if that
makes sense in the future.  
>> That could work, although if the end direction is something akin to the
>> above, then it may make less sense to do this as an intermediate step.
>> If not, then it shouldn''t be too hard to do. 
>
>I''m not 100% sure what you mean by end direction?  Do you mean the
change
>to a probe only enumeration call?  I think that would be a simple change
>since the call would be very similar on the client side, but it would
>allow the backend to skip any changes which are only required when the
>device is actually brought up.
Yes, changing to probe only doing the enumeration of devices.  And then,
loading the driver really only has to connect to the devices and set
their status.  You''re right on it not being that difficult once I got a
better feeling of my way around the maze of calls in xend and how they
interact.  

This also adds the advantage that instead of having to do a complete
rescan for adding a new device (right now, adding a device ends up
sending an INTERFACE_CHANGED message and the burden is then on the
frontend drivers to determine exactly what that entails), you can just
send a message that says "new block device, index 2" and that gets
matched to a driver and the driver only needs to change things for that
device.
>I think you want to query each backend and find out what devices it has.
>I''m not sure if the enumeration function should be generic or
>device-specific.  Our current device-specific approach could probably be
>changed to be generic but in a first instance it would be quickest to
>implement this using the existing "queries" since there are only
two
>device types.
Well, there are three now with USB.  And requiring the guest kernels to
know it all seems silly.  It''s actually pretty simple to do in a
generic
fashion.
>Have you made any progress on this?
Not as much as I would _like_ to have, just due to some other things
keeping me busy but I am making slow but steady progress.  My plan is to
try to work on it some more tonight and hopefully get a next iteration
out for more comments in the next day or so.

Jeremy



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> >I think you want to query each backend and find out what devices it
has.
> >I''m not sure if the enumeration function should be generic or
> >device-specific.  Our current device-specific approach could probably
be
> >changed to be generic but in a first instance it would be quickest to
> >implement this using the existing "queries" since there are
only two
> >device types.
> 
> Well, there are three now with USB.  And requiring the guest kernels to
> know it all seems silly.  It''s actually pretty simple to do in a
generic
> fashion.
I thought that the issue of enumerating interfaces and supporting
multiple backends was one of the major reasons that we were looking at
restructuring the control tools.  I think it makes a whole lot more
sense to have the list of devices and related state for each domain in
a persistent store/registry in dom0, primarily because it gives us a
point of indirection to allow backend restarts/hotswaps and means
other control tools can be added to look at the same state without
modifying the drivers or control messages.

This is vaguely what I think domain startup should look like with new
tools.  Bear in mind that it is quarter to nine on a saturday morning.
 These probably need a bit more thought, but I think these are along
the lines of what I have talked to keir about in the past.  Setting up
net/block interfaces for a new domain looks something like:

1. During domain creation, xend pushes a list or interfaces under the
new domain''s part of the registry:

   /domn/blkifs/0/[each bit of state]

2. xend kicks each associated backend interface with an
FE_CREATE(domid) control message.  Each backend driver then scans the
registry for interfaces under its control and registers them.

3. As the domain starts up, its driver probes the registry for
enumeration, and then wires up interfaces based on the details there.

This is clearly more vague than it should be.  If anthony''s new C
control tools are sufficiently functional, then maybe we can make some
fast progress in this direction though.  In general I like breaking
out state management because it gets us away from per-driver protocols
to manage simple state across front and backend.  It might even be
nice to see if we could come up with a clear general mechanism for
signalling shared memory/event channel setup along these lines as
well.

a.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> to manage simple state across front and backend.  It might even be
> nice to see if we could come up with a clear general mechanism for
> signalling shared memory/event channel setup along these lines as
> well.
The current setup where each driver replicates all the device channel setup 
code code is a bit nasty.

It would be well worth the effort to split out the device channel protocols 
into a device-specific portion (details of associated VBDs / VIFs / feather 
dusters) and a generic portion (details about the shared memory page, event 
channels, etc).

Mark
(who has recently added one device class and is now working on another)


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel