I''ve been looking at the next-generation of control tools, and see some areas that I think still need some work. I''m working on some code that will be ready "soon", but I wanted to post and get feedback on my ideas now rather than later. Here are the things I''m working on: 1. Providing a higher-level consistent API for domain actions (create, save, migrate, restore, pause, etc). 2. Making the tools/libraries agnostic w.r.t. the guest OS type. 3. Pushing rate-limiting lower into the IO streams. 4. Making xfrd''s functionality available as a library. I''ll give you some of my motivation, for each of these items: 1. Higher-level API: This is listed as a task in Andrew/Keir/Christian''s Xen Control Tools Design Plan. This is also becoming important to support items #2 and #4. 2. OS agnostic: I can give some examples of where this is not the case (migrations assume Linux; rate limiting is only Linux, etc). But beyond specific examples, making things OS agnostic forces some concepts to be clarified in the code. High-level commands (such as "create") logically have a fixed sequence of steps, regardless of the OS type. Those high-level logical steps should be separate from OS-specific implementation details. Mixing the two creates maintenance and portability headaches. 3. Rate limiting: Rate limiting should be available to all IO streams, regardless of OS and regardless of the current action. Currently it is only available for a Linux domain. I''m pushing this down the stack, so that it is more transparent (but optional still). 4. libxfrd: Xend and xfrd are tied pretty tightly together via an SXP-based protocol. Once xend is gone, either xfrd will need to be reworked or the xend replacement will need to adhere to this protocol. Since we''re moving towards multiple single-purpose tools (e.g., VM-tools), continuing to maintain this protocol would be awkward. I would prefer to have xfrd''s functionality available as a library, that any tool can link against. I still see the receiving end of xfrd running as a daemon (a small daemon using the xfrd library, that is). But commands that want to initiate xfrd-style sends can call the library directly. Other advantages of this are that progress reports can be done via a simple C callback, and you can get finer-grained error reporting (rather than the current success/failure code). Once you start looking at this within a larger management framework, these things become important. I''m refactoring / rewriting / writing code in xutil, libxc, and xfrd (although it wouldn''t be hard to slip libxen in there instead of libxc if that is the ultimate direction). I hope to have something to show in a week or two. Cheers, Charles _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote:> 1. Providing a higher-level consistent API for domain actions > (create, save, migrate, restore, pause, etc).I''m no longer convinced there is a much higher API to have. Except for create and migrate, all the actions you list are already single functions in libxc, migrate is save | restore. It might be useful to group several functions together for create, but it''s not quite clear how/where device configuration fits in there.> I''m refactoring / rewriting / writing code in xutil, libxc, and xfrd > (although it wouldn''t be hard to slip libxen in there instead of libxc > if that is the ultimate direction). I hope to have something to show > in a week or two.Please note that xfrd does no longer exist in -unstable. Also we don''t use libxutil anymore. Xend handles the first half of a relocation itself and then runs the xc_save or xc_restore helper programs to do the second part. During the first part of a relocation, the xend domain configuration is exchanged and the format of this part is specific to xend. The xc_save and xc_restore helpers are merely wrappers for the xc_linux_save and xc_linux_restore functions and use pipes to communicate with xend and write/read the virtual machine image to/from a file handle or socket. christian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian, I''ve done a pull and now see the functionality in XendCheckpoint.py; thanks for the heads-up. But how do you see things outside of xm & xend accessing this functionality? In particular, I''m thinking of a CIMOM provider written in C++. Forking/exec-ing an "xm migrate" command is less than ideal, for several reasons (progress reporting, error reporting without having to grok text, overhead on a busy server, ...) In my ideal world, this level of functionality would be in C or C++ libraries, so you can put whatever you want on top of it, be it Python commands or C++ CIMOM code or anything else. I don''t necessarily want to revive the old Python debate, but this does complicate things. Thanks, Charles>>>christian.limpach@gmail.com 05/31/05 3:07 pm >>>On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote:>1. Providing a higher-level consistent API for domain actions >(create, save, migrate, restore, pause, etc).I''m no longer convinced there is a much higher API to have. Except for create and migrate, all the actions you list are already single functions in libxc, migrate is save | restore. It might be useful to group several functions together for create, but it''s not quite clear how/where device configuration fits in there.>I''m refactoring / rewriting / writing code in xutil, libxc, and xfrd >(although it wouldn''t be hard to slip libxen in there instead of libxc >if that is the ultimate direction). I hope to have something to show >in a week or two.Please note that xfrd does no longer exist in -unstable. Also we don''t use libxutil anymore. Xend handles the first half of a relocation itself and then runs the xc_save or xc_restore helper programs to do the second part. During the first part of a relocation, the xend domain configuration is exchanged and the format of this part is specific to xend. The xc_save and xc_restore helpers are merely wrappers for the xc_linux_save and xc_linux_restore functions and use pipes to communicate with xend and write/read the virtual machine image to/from a file handle or socket. christian Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 31 May 2005 14:29, Charles Coffing wrote:> Christian, > > I''ve done a pull and now see the functionality in XendCheckpoint.py; thanks > for the heads-up. > > But how do you see things outside of xm & xend accessing this > functionality? In particular, I''m thinking of a CIMOM provider written in > C++. Forking/exec-ing an "xm migrate" command is less than ideal, for > several reasons (progress reporting, error reporting without having to grok > text, overhead on a busy server, ...) In my ideal world, this level of > functionality would be in C or C++ libraries, so you can put whatever you > want on top of it, be it Python commands or C++ CIMOM code or anything > else. > > I don''t necessarily want to revive the old Python debate, but this does > complicate things.Hi Charles, Please look at vm-tools, a set of C tools for managing Xen systems. It''s a work in progress, but it''s coming along. We have single purpose C tools that can easily be used to do what you require, whether wrapping them with scripts or accessing them directly. Here''s Anthony''s post mentioned the latest version: http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01080.html Here''s where to get it: http://www.cs.utexas.edu/users/aliguori/vm-tools/ Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>But how do you see things outside of xm & xend accessing this functionality? >In particular, I''m thinking of a CIMOM provider written in C++. >Forking/exec-ing an "xm migrate" command is less than ideal, for several >reasons (progress reporting, error reporting without having to grok text, >overhead on a busy server, ...) In my ideal world, this level of >functionality would be in C or C++ libraries, so you can put whatever you >want on top of it, be it Python commands or C++ CIMOM code or anything >else.The basic save/restore stuff is now simplified (in particular has no dependencies on a callback mechanism out of the library). To do e.g. a non-live migrate you then effectively need to: * work out what the config of the domain is - do a save - use the config to create an ''empty'' domain - restore into this, and * perform final configuration tweaks The key thing is that the steps with ''*'' (at least) require config information which xen doesn''t have. You can maintain this yourself in your C++ app or, in the new tools world (in progress), get this from ''xenstore''. There''s no real short-cut around this, and it has very little to do with python. There is a real + desired separation of concerns between config info that xen needs to know about (domain instance state) and config info that a ''user'' might care about (VM state). This latter info is currently managed by xend (one of its primary functions) but could equally be managed by the vmtools daemon or an arbitrary user app. However since in most scenarios we believe it''s sane to keep all this consistent, having a single locus for such config info (viz. xenostore) makes a lot of sense. Hope this addresses your concerns, cheers, S. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Coffing wrote:> 2. Making the tools/libraries agnostic w.r.t. the guest OS type.I recently created a small Xen guest that is able to build a guest OS from a ramdisk containing a stage2 loader and a kernel image. My goal is to have it load from a block device or the network as well as from the ram disk. Starting my guest instead of vmlinuz directly would allow unification of domain builders from the dom0 perspective, and would prevent ELF-parsing attacks on dom0. The same guest would be used for reviving checkpoints, so there would not need to be a special linux_load (linux_save can be handled using self-checkpointing to a raw device btw). If there is interest I can post the source. Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote:> But how do you see things outside of xm & xend accessing > this functionality? In particular, I''m thinking of a CIMOM > provider written in C++. Forking/exec-ing an "xm migrate" > command is less than ideal, for several reasons (progress > reporting, error reporting without having to grok text, overhead > on a busy server, ...) In my ideal world, this level of > functionality would be in C or C++ libraries, so you can put > whatever you want on top of it, be it Python commands or > C++ CIMOM code or anything else.Well, the non-xend specific code needed to do a relocation is in libxc[1] and I believe IBM''s vm-tools also use this code. This code only saves/restores the low-level virtual machine image, it doesn''t even know about checkpointing/relocation, i.e. how the data gets on or off the disk or to a different machine or how to configure the devices needed by the virtual machine on the machine where the virtual machine is restored. All this configuration management and setup work needs to be done by a tool like xend or a toolset like vm-tools. xend currently handles storage and management of configuration information and configuration of devices according to this information and on top provides a client/server interface to control it all[2]. In a next step, we''re going to move the storage and configuration parts into what we call ''xenstore''. With xenstore it will be even easier to write simple command line tools or interface from C or C++ libraries, all the gory details of device setup will be hidden. We hope that with this seperation xend will be reduced to an even more reasonable size and that there won''t be a need for alternate toolsets, or that these would at least use xenstore. christian [1] I think I''ll make the control pipe and error/progress reporting filehandles function call arguments to make it easier to use xc_linux_{save,restore} when you link libxc into your application. Although I really think you want to run these in a seperate process to isolate them from your main application. If there''s real demand we could bring back something like ioctxt to give you callbacks for control/error/progress. [2] and juggling around console data. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/31/05, Steven Hand <Steven.Hand@cl.cam.ac.uk> wrote:> The basic save/restore stuff is now simplified (in particular has no > dependencies on a callback mechanism out of the library). To do e.g. > a non-live migrate you then effectively need to: > > * work out what the config of the domain is > - do a save > - use the config to create an ''empty'' domain > - restore into this, and > * perform final configuration tweaksThere''s actually still a callback during save, when we need to tell the domain to suspend itself. This uses the control channel between xend and the domain to send a control message to the domain. We could maybe make xc_linux_save talk directly to the domain through xcs or we could use xenstore for suspend requests... christian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I am interested in your code. I don''t know if you will be posting the source to the list, but if you decide not to could you at least send it to me? I am currently working on trusted boot (ie TPM support, measuring new domains etc., we were also considering moving the domain builder out of domain0, to a separate domain maybe, I am not quite certain how your builder works, but it looks to be quite a reasonable approach). Cheers Gregor> Charles Coffing wrote: > > 2. Making the tools/libraries agnostic w.r.t. the guest OS type. > > I recently created a small Xen guest that is able to build a guest OS > from a ramdisk containing a stage2 loader and a kernel image. My goal is > to have it load from a block device or the network as well as from the > ram disk. Starting my guest instead of vmlinuz directly would allow > unification of domain builders from the dom0 perspective, and would > prevent ELF-parsing attacks on dom0. The same guest would be used for > reviving checkpoints, so there would not need to be a special linux_load > (linux_save can be handled using self-checkpointing to a raw device btw). > > If there is interest I can post the source. > > Jacob > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Quidquid latine dictum sit, altum viditur --- Anon _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian,> [1] I think I''ll make the control pipe and error/progress reporting > filehandles function call arguments to make it easier to use > {save,restore} when you link libxc into your application. > Although I really think you want to run these in a seperate process to > isolate them from your main application. If there''s real demand we > could bring back something like ioctxt to give you callbacks for > control/error/progress.Is there a specific reason you suggest running these in a separate process? I''m wondering if this is just a general preference because isolation tends to be a good thing, or if you suspect, say, a maliciously crafted domain could somehow crash the domain builder function. Thanks for your explanations, by the way. Charles _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jun 01, 2005 at 07:56:54AM -0600, Charles Coffing wrote:> > [1] I think I''ll make the control pipe and error/progress reporting > > filehandles function call arguments to make it easier to use > > {save,restore} when you link libxc into your application. > > Although I really think you want to run these in a seperate process to > > isolate them from your main application. If there''s real demand we > > could bring back something like ioctxt to give you callbacks for > > control/error/progress. > > Is there a specific reason you suggest running these in a separate > process? I''m wondering if this is just a general preference because > isolation tends to be a good thing, or if you suspect, say, a > maliciously crafted domain could somehow crash the domain builder > function.I suspect not all error paths to be well tested. christian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Coffing wrote:>Christian, > >I''ve done a pull and now see the functionality in XendCheckpoint.py; thanks for the heads-up. > >But how do you see things outside of xm & xend accessing this functionality? In particular, I''m thinking of a CIMOM provider written in C++. Forking/exec-ing an "xm migrate" command is less than ideal, for several reasons (progress reporting, error reporting without having to grok text, overhead on a busy server, ...) In my ideal world, this level of functionality would be in C or C++ libraries, so you can put whatever you want on top of it, be it Python commands or C++ CIMOM code or anything else. > >I''m quite happy with the direction the lower level infrastructure is going in. Once the store gets straightened out, the control channel multiplexing daemon becomes less interesting (which is the biggest barrier to tools entry today). Christian''s latest modifications to the xc_save/restore code were fantastic and we were able to start using it quickly. I''d still like to see some of the libxc interfaces refactored a bit (for instance, I''d like xc_domain_create to do a bit less so better error conditions can be obtained) but I think we''re going to try to start helping to improve the existing error paths first. I''ve always thought of VM-Tools as a mechanism to determine what aspects of Xen needs more work to support additional management tools. Your point about OS-agnostic interfaces is a good one--there''s a bit of a combinatorial domain creation problem as we support more types of domains on more types of architecture (and even if we ever support things other than Linux for dom0). Regards, Anthony Liguori>I don''t necessarily want to revive the old Python debate, but this does complicate things. > >Thanks, >Charles > > > > >>>>christian.limpach@gmail.com 05/31/05 3:07 pm >>> >>>> >>>> >On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote: > > > >>1. Providing a higher-level consistent API for domain actions >>(create, save, migrate, restore, pause, etc). >> >> > >I''m no longer convinced there is a much higher API to have. Except >for create and migrate, all the actions you list are already single >functions in libxc, migrate is save | restore. It might be useful to >group several functions together for create, but it''s not quite clear >how/where device configuration fits in there. > > > >>I''m refactoring / rewriting / writing code in xutil, libxc, and xfrd >>(although it wouldn''t be hard to slip libxen in there instead of libxc >>if that is the ultimate direction). I hope to have something to show >>in a week or two. >> >> > >Please note that xfrd does no longer exist in -unstable. Also we >don''t use libxutil anymore. >Xend handles the first half of a relocation itself and then runs the >xc_save or xc_restore helper programs to do the second part. During >the first part of a relocation, the xend domain configuration is >exchanged and the format of this part is specific to xend. The >xc_save and xc_restore helpers are merely wrappers for the >xc_linux_save and xc_linux_restore functions and use pipes to >communicate with xend and write/read the virtual machine image to/from >a file handle or socket. > > christian > > >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel