Daniel P. Berrange wrote:> On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > >> I''m looking into creating a driver for the new Xen xl/libxl toolstack >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 >> release. >> >> My first hurdle is deciding whether this should be a new driver or >> integrated with existing xen-unified driver. Initially I thought a new >> driver would be a better approach - a clean break from the old code, >> similar to the xenapi driver. libxenlight is also stateless (no managed >> domains), which seems like another good argument for a new driver. But >> libxenlight is really just another interface into the same hypervisor, >> so in that regard it should be a xen-unified subdriver. >> > > Something on the system must be stateful, continually monitoring > guests & taking neccessary actions ? eg If XenD isn''t used, then > what is responsible for restarting guests which crash, or performing > core dumps on crashed guests, etc, etc ? >Good questions. I have just started looking at the new toolstack, and frankly don''t yet know how this is handled. Adding xen-devel for comment ...> This would have a bearing on how best to design a libvirt driver > > >> There are certainly benefits to the xen-unified subdriver approach, e.g. >> the existing hypervisor and xenstore subdrivers can be leveraged, the >> former providing all the capabilities code. But AFAIK, libxenlight and >> xend should not be used together, so I don''t think we would want the >> xend subdriver activated if libxenlight is detected. Supposedly xl can >> be used as a direct replacement for xm, allowing unconditional use of >> that subdriver. >> >> BTW, Ian Jackson responded [2] to some of my questions regarding >> compatibility between old and new toolstack if you are interested. >> >> I''d like to hear other''s opinions on a new driver vs. a xen-unified >> subdriver. >> > > Due to the number of revisions of Xen userspace stack, and the > need to talk to so many pieces to get an efficient driver, the > current Xen unified driver is rather hairy. Particularly if > XenD itself is deprecated as a control mechanism, then I''d > go for a new standalone driver, that runs from libvirtd context > and leverages the standard libvirt storage/network/inteface > drivers for non-HV stuff (which I assume libxenlight doesn''t > cover). >Correct. AFAICT, libxenlight does not cover any host storage or network management. A libxenlight driver will be a hypervsisor driver only. Regards, Jim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Gianni Tedesco
2011-Jan-21 16:33 UTC
Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver
On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote:> Daniel P. Berrange wrote: > > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > > > >> I''m looking into creating a driver for the new Xen xl/libxl toolstack > >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 > >> release. > >> > >> My first hurdle is deciding whether this should be a new driver or > >> integrated with existing xen-unified driver. Initially I thought a new > >> driver would be a better approach - a clean break from the old code, > >> similar to the xenapi driver. libxenlight is also stateless (no managed > >> domains), which seems like another good argument for a new driver. But > >> libxenlight is really just another interface into the same hypervisor, > >> so in that regard it should be a xen-unified subdriver. > >> > > > > Something on the system must be stateful, continually monitoring > > guests & taking neccessary actions ? eg If XenD isn''t used, then > > what is responsible for restarting guests which crash, or performing > > core dumps on crashed guests, etc, etc ? > > > > Good questions. I have just started looking at the new toolstack, and > frankly don''t yet know how this is handled. Adding xen-devel for > comment ...Yes and no, in xen, the hypervisor internal data structures combined with xenstore is the state (pretty much anyway). The purpose of libxenlight is to abstract away these details. However, when a domain is created with libxenlight one must wait for a few events to ensure the proper clean-up of the domain. The events are basically ''domain death'' or ''device ejected''. These exist so the toolstack can handle rebooting, core dumping the domain, inserting the next CD image, or whatever. I am not sure how libvirt handles such events normally but you may either integrate with a poll/select based mainloop or have threads or processes with self-pipes handling these. So in summary, state is maintained by the xen system outside of any libvirt driver, but handling the kind of events you mention will require some special work to be done for domains created by libvirt.> > This would have a bearing on how best to design a libvirt driver > > > > > >> There are certainly benefits to the xen-unified subdriver approach, e.g. > >> the existing hypervisor and xenstore subdrivers can be leveraged, the > >> former providing all the capabilities code. But AFAIK, libxenlight and > >> xend should not be used together, so I don''t think we would want the > >> xend subdriver activated if libxenlight is detected. Supposedly xl can > >> be used as a direct replacement for xm, allowing unconditional use of > >> that subdriver. > >> > >> BTW, Ian Jackson responded [2] to some of my questions regarding > >> compatibility between old and new toolstack if you are interested. > >> > >> I''d like to hear other''s opinions on a new driver vs. a xen-unified > >> subdriver. > >> > > > > Due to the number of revisions of Xen userspace stack, and the > > need to talk to so many pieces to get an efficient driver, the > > current Xen unified driver is rather hairy. Particularly if > > XenD itself is deprecated as a control mechanism, then I''d > > go for a new standalone driver, that runs from libvirtd context > > and leverages the standard libvirt storage/network/inteface > > drivers for non-HV stuff (which I assume libxenlight doesn''t > > cover). > > > > Correct. AFAICT, libxenlight does not cover any host storage or network > management. A libxenlight driver will be a hypervsisor driver only.Yes definitely. We handle attaching and detaching NIC''s and block devices. Any other management is orthogonal to libxenlight (as it is for xend). BTW. I would also add my vote for a separate code-base for xend/traditional vs. libxenlight drivers as there is no similarity of interface. The xend HTTP RPC calls are unrelated to the libxl API. In any case, please keep xen-devel in the distribution list :) Gianni _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2011-Jan-21 16:44 UTC
Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver
On Fri, Jan 21, 2011 at 04:33:55PM +0000, Gianni Tedesco wrote:> On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote: > > Daniel P. Berrange wrote: > > > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > > > > > >> I''m looking into creating a driver for the new Xen xl/libxl toolstack > > >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 > > >> release. > > >> > > >> My first hurdle is deciding whether this should be a new driver or > > >> integrated with existing xen-unified driver. Initially I thought a new > > >> driver would be a better approach - a clean break from the old code, > > >> similar to the xenapi driver. libxenlight is also stateless (no managed > > >> domains), which seems like another good argument for a new driver. But > > >> libxenlight is really just another interface into the same hypervisor, > > >> so in that regard it should be a xen-unified subdriver. > > >> > > > > > > Something on the system must be stateful, continually monitoring > > > guests & taking neccessary actions ? eg If XenD isn''t used, then > > > what is responsible for restarting guests which crash, or performing > > > core dumps on crashed guests, etc, etc ? > > > > > > > Good questions. I have just started looking at the new toolstack, and > > frankly don''t yet know how this is handled. Adding xen-devel for > > comment ... > > Yes and no, in xen, the hypervisor internal data structures combined > with xenstore is the state (pretty much anyway). The purpose of > libxenlight is to abstract away these details. > > However, when a domain is created with libxenlight one must wait for a > few events to ensure the proper clean-up of the domain. The events are > basically ''domain death'' or ''device ejected''. These exist so the > toolstack can handle rebooting, core dumping the domain, inserting the > next CD image, or whatever. I am not sure how libvirt handles such > events normally but you may either integrate with a poll/select based > mainloop or have threads or processes with self-pipes handling these. > > So in summary, state is maintained by the xen system outside of any > libvirt driver, but handling the kind of events you mention will require > some special work to be done for domains created by libvirt.Ok, so basically the mgmt app that is using the libxenlight API is taking the place of XenD. Thus in the new scheme libvirtd would have to handle those kind of events. This is fine because we already do that kind of thing with our other HV drivers for QEMU/KVM, UserModeLinux and LXC.> > > > This would have a bearing on how best to design a libvirt driver > > > > > > > > >> There are certainly benefits to the xen-unified subdriver approach, e.g. > > >> the existing hypervisor and xenstore subdrivers can be leveraged, the > > >> former providing all the capabilities code. But AFAIK, libxenlight and > > >> xend should not be used together, so I don''t think we would want the > > >> xend subdriver activated if libxenlight is detected. Supposedly xl can > > >> be used as a direct replacement for xm, allowing unconditional use of > > >> that subdriver. > > >> > > >> BTW, Ian Jackson responded [2] to some of my questions regarding > > >> compatibility between old and new toolstack if you are interested. > > >> > > >> I''d like to hear other''s opinions on a new driver vs. a xen-unified > > >> subdriver. > > >> > > > > > > Due to the number of revisions of Xen userspace stack, and the > > > need to talk to so many pieces to get an efficient driver, the > > > current Xen unified driver is rather hairy. Particularly if > > > XenD itself is deprecated as a control mechanism, then I''d > > > go for a new standalone driver, that runs from libvirtd context > > > and leverages the standard libvirt storage/network/inteface > > > drivers for non-HV stuff (which I assume libxenlight doesn''t > > > cover). > > > > > > > Correct. AFAICT, libxenlight does not cover any host storage or network > > management. A libxenlight driver will be a hypervsisor driver only. > > Yes definitely. We handle attaching and detaching NIC''s and block > devices. Any other management is orthogonal to libxenlight (as it is for > xend). > > BTW. I would also add my vote for a separate code-base for > xend/traditional vs. libxenlight drivers as there is no similarity of > interface. The xend HTTP RPC calls are unrelated to the libxl API.Agreed, now you clarified my question(s) above, I think a dedicated libxenlight driver in libvirtd is the only sensible option. On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future. Regards, Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-21 16:54 UTC
Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver
On Fri, 21 Jan 2011, Jim Fehlig wrote:> Daniel P. Berrange wrote: > > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > > > >> I''m looking into creating a driver for the new Xen xl/libxl toolstack > >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 > >> release. > >> > >> My first hurdle is deciding whether this should be a new driver or > >> integrated with existing xen-unified driver. Initially I thought a new > >> driver would be a better approach - a clean break from the old code, > >> similar to the xenapi driver. libxenlight is also stateless (no managed > >> domains), which seems like another good argument for a new driver. But > >> libxenlight is really just another interface into the same hypervisor, > >> so in that regard it should be a xen-unified subdriver. > >> > > > > Something on the system must be stateful, continually monitoring > > guests & taking neccessary actions ? eg If XenD isn''t used, then > > what is responsible for restarting guests which crash, or performing > > core dumps on crashed guests, etc, etc ? > > > > Good questions. I have just started looking at the new toolstack, and > frankly don''t yet know how this is handled. Adding xen-devel for > comment ...xenstore is stateful and the hypervisor keeps a state of the system so there is no need for libxenlight to maintain yet another additional state of the system on top of them: it just relies on the informations already available through them. xl (the minimal C toolstack on top of libxenlight) spawns a daemon per domain to handle domain death events and cdrom events. I should also mention that one of the design goals of libxenlight is that no libxenlight users (toolstacks like libvirt) should need to use anything else but libxenlight to perform Xen operations. So you shouldn''t need to call any xenctrl functions or deal with xenstore directly.> > This would have a bearing on how best to design a libvirt driverI don''t know much about libvirt but I would write a brand new libxenlight driver.> >> There are certainly benefits to the xen-unified subdriver approach, e.g. > >> the existing hypervisor and xenstore subdrivers can be leveraged, the > >> former providing all the capabilities code. But AFAIK, libxenlight and > >> xend should not be used together, so I don''t think we would want the > >> xend subdriver activated if libxenlight is detected. Supposedly xl can > >> be used as a direct replacement for xm, allowing unconditional use of > >> that subdriver. > >> > >> BTW, Ian Jackson responded [2] to some of my questions regarding > >> compatibility between old and new toolstack if you are interested. > >> > >> I''d like to hear other''s opinions on a new driver vs. a xen-unified > >> subdriver. > >> > > > > Due to the number of revisions of Xen userspace stack, and the > > need to talk to so many pieces to get an efficient driver, the > > current Xen unified driver is rather hairy. Particularly if > > XenD itself is deprecated as a control mechanism, then I''d > > go for a new standalone driver, that runs from libvirtd context > > and leverages the standard libvirt storage/network/inteface > > drivers for non-HV stuff (which I assume libxenlight doesn''t > > cover). > > > > Correct. AFAICT, libxenlight does not cover any host storage or network > management. A libxenlight driver will be a hypervsisor driver only.Yep. Feel free to ask any questions you might have on libxenlight to xen-devel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:> > On a related note, where does XenAPI fit into this picture ? > Previously XenD would provide (at least some of) the XenAPI > functionality in the open source stack, while I understand > that the XenEnterprise had a separate impl of XenAPI. It > sounds like XenAPI is likely only relevant for XenEnterprise > connectivity in the future.The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer). Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-21 17:01 UTC
Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver
On Fri, 21 Jan 2011, Daniel P. Berrange wrote:> Ok, so basically the mgmt app that is using the libxenlight API > is taking the place of XenD. Thus in the new scheme libvirtd > would have to handle those kind of events.Right. Libxenlight provides all the functions needed to do that and you can find a reference implementation in xl (tools/libxl/xl_cmdimpl.c).> This is fine because > we already do that kind of thing with our other HV drivers for > QEMU/KVM, UserModeLinux and LXC.Good to hear.> On a related note, where does XenAPI fit into this picture ? > Previously XenD would provide (at least some of) the XenAPI > functionality in the open source stack, while I understand > that the XenEnterprise had a separate impl of XenAPI. It > sounds like XenAPI is likely only relevant for XenEnterprise > connectivity in the future.Nowadays XAPI (the XenServer toolstack) is Open Source and provides a full feature implementation of the API. I think that users that need XenAPI functionalities are probably best served by using XAPI rather than XenD. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell wrote:> On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote: > >> On a related note, where does XenAPI fit into this picture ? >> Previously XenD would provide (at least some of) the XenAPI >> functionality in the open source stack, while I understand >> that the XenEnterprise had a separate impl of XenAPI. It >> sounds like XenAPI is likely only relevant for XenEnterprise >> connectivity in the future. >> > > The xapi toolstack which implements XenAPI is now open source as well > and is used in XCP (as well as XenServer). >But there is no intent on putting this toolstack in the traditional Xen releases from xen.org correct? Jim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-01-21 at 17:02 +0000, Jim Fehlig wrote:> Ian Campbell wrote: > > On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote: > > > >> On a related note, where does XenAPI fit into this picture ? > >> Previously XenD would provide (at least some of) the XenAPI > >> functionality in the open source stack, while I understand > >> that the XenEnterprise had a separate impl of XenAPI. It > >> sounds like XenAPI is likely only relevant for XenEnterprise > >> connectivity in the future. > >> > > > > The xapi toolstack which implements XenAPI is now open source as well > > and is used in XCP (as well as XenServer). > > > > But there is no intent on putting this toolstack in the traditional Xen > releases from xen.org correct?Although XCP and xapi are themselves xen.org projects they are maintained and released separately. The intention is that in the future xapi will use libxl too, but still as an external consumer not part of the main xen hg repository. The xcp and xapi folks hang out on the xen-api@lists.xensource.com list. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Gianni Tedesco
2011-Jan-21 17:19 UTC
Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver
On Fri, 2011-01-21 at 17:01 +0000, Stefano Stabellini wrote:> On Fri, 21 Jan 2011, Daniel P. Berrange wrote: > > Ok, so basically the mgmt app that is using the libxenlight API > > is taking the place of XenD. Thus in the new scheme libvirtd > > would have to handle those kind of events. > > Right. > Libxenlight provides all the functions needed to do that and you can > find a reference implementation in xl (tools/libxl/xl_cmdimpl.c). > > > > This is fine because > > we already do that kind of thing with our other HV drivers for > > QEMU/KVM, UserModeLinux and LXC. > > Good to hear.+1> > On a related note, where does XenAPI fit into this picture ? > > Previously XenD would provide (at least some of) the XenAPI > > functionality in the open source stack, while I understand > > that the XenEnterprise had a separate impl of XenAPI. It > > sounds like XenAPI is likely only relevant for XenEnterprise > > connectivity in the future. > > Nowadays XAPI (the XenServer toolstack) is Open Source and provides a > full feature implementation of the API. I think that users that need > XenAPI functionalities are probably best served by using XAPI rather > than XenD.Yes agreed, in my personal opinion, XenAPI in xend (and perhaps xend itself) has a perilous future. Gianni _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jim Fehlig writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight
driver"):> Ian Campbell wrote:
> > The xapi toolstack which implements XenAPI is now open source as well
> > and is used in XCP (as well as XenServer).
>
> But there is no intent on putting this toolstack in the traditional Xen
> releases from xen.org correct?
That''s right, there isn''t.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stefano Stabellini writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight
driver"):> I should also mention that one of the design goals of libxenlight is
> that no libxenlight users (toolstacks like libvirt) should need to use
> anything else but libxenlight to perform Xen operations. So you
> shouldn''t need to call any xenctrl functions or deal with xenstore
> directly.
This is quite important. We have tried to do this and we have
succeeded for the "xl" command''s requirements, at least.
However since we currently have only one user of libxl, "xl" itself,
it is very likely that you will find that there are deficiencies in
the libxl API which make it difficult to use properly in libvirt.
We would very much like to fix these problems so please report them
all to us. We will work with you to make libxl have the API you need
so that you can write libvirt properly.
Inevitably this will mean that the libvirt adaptation will need the
libxl from Xen 4.2, not the Xen 4.1 version which is currently in
freeze. However in Xen 4.1 xend is still supported.
Please also send your libvirt patches to xen-devel for review, etc.
> I don''t know much about libvirt but I would write a brand new
> libxenlight driver.
Yes, absolutely. Having taken a look at the existing code in libvirt
for driving xenstore/xen/etc., none of it looks sensibly reuseable.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel