Dan Magenheimer
2009-Jun-17 23:00 UTC
[Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
Is there any documentation on the format of the file/stream used for save/restore/migrate? Or is it just a sequence of chunks of bytes pre-defined only by the ordering of the save/restore code? I found this 2006 xen-devel thread, but it doesn''t look like it (other than hvm additions) ever happened? E.g. still no versioning, self-identification, extensibility? http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.html I''m starting to look at save/restore/migrate for tmem and will need to communicate the following information via file/stream: - pool id and characteristics (a small number of bytes of data), for some small number of pools - for some classes of pools, some number of "pages" of data, each page consisting of a "handle" (128 bits) and PAGE_SIZE bytes of data associated with that handle - for some of these pages of data, a handle+invalidate (see below) - (optional) in some cases the pages will be pre-compressed; each can be decompressed on the source side and recompressed on the destination side, but this seems a sad waste of cpu cycles (though necessary if the compression algorithm were different on source and destination); if possible, save/transmit non-decompressed data Note that for the pages of data, dirtying during migration is not possible, however invalidation IS possible. E.g. unlike normally addressable pages which may be transmitted multiple times during a live migration, a transmitted tmem page (handle and data) will be transmitted only once, but may be followed at some point with a handle+invalidate. The ordering of tmem info/pages vs current saved info/data is flexible, but the number of pages could be very large, so for live migrate, transmission should NOT be postponed until the "final pass" of normal page transmission (e.g. after the domain has been paused on the source machine). I also need code to verify that the destination has tmem support and it is enabled. Only PV domains can use tmem so no HVM changes should be necessary. Any pointers or suggestions welcome, especially any thoughts on changes that might be required above libxc such as in python code or (heaven forbid) ioemu/qemu. Thanks for any help! Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2009-Jun-18 00:17 UTC
[Xen-devel] Re: save image file format? and [RFC] tmem save/restore/migrate
On Wed, Jun 17, 2009 at 04:00:30PM -0700, Dan Magenheimer wrote:> Is there any documentation on the format of the file/stream > used for save/restore/migrate? Or is it just a sequence of > chunks of bytes pre-defined only by the ordering of the > save/restore code? > > http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.htmlIt''s still the case, sadly. The HVM changes were hacked in instead. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2009-Jun-18 08:51 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
At 00:00 +0100 on 18 Jun (1245283230), Dan Magenheimer wrote:> Is there any documentation on the format of the file/stream > used for save/restore/migrate? Or is it just a sequence of > chunks of bytes pre-defined only by the ordering of the > save/restore code? > > I found this 2006 xen-devel thread, but it doesn''t look like > it (other than hvm additions) ever happened? E.g. still > no versioning, self-identification, extensibility?It''s still the ad-hoc stream-of-data format, I''m afraid. The whole format needs a good kicking. It''s not even portable between 32-bit and 64-bit tools. Gianluca (Cc''d) is just starting to look at the save/restore code in the hopes of making it more sane, so now would be a good time to bring up any suggestions. The usual way of adding new fields is to grab another negative number in the length-of-the-next-block-of-frames field. BTW, the HVM save records supplied by Xen to libxc _are_ self-identifying and extensible (and there''s room in the header for a version number, though by sticking to the principle of only transferring architectural state we''ve avoided the need to use it so far). But they''re just dropped into the stream after the memory pages and before xend glues on the qemu record. Cheers, Tim.> http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.html > > I''m starting to look at save/restore/migrate for tmem and will > need to communicate the following information via file/stream: > > - pool id and characteristics (a small number of bytes of data), > for some small number of pools > - for some classes of pools, some number of "pages" of data, > each page consisting of a "handle" (128 bits) and PAGE_SIZE > bytes of data associated with that handle > - for some of these pages of data, a handle+invalidate (see below) > - (optional) in some cases the pages will be pre-compressed; > each can be decompressed on the source side and recompressed > on the destination side, but this seems a sad waste of > cpu cycles (though necessary if the compression algorithm > were different on source and destination); if possible, > save/transmit non-decompressed data > > Note that for the pages of data, dirtying during migration > is not possible, however invalidation IS possible. E.g. > unlike normally addressable pages which may be transmitted > multiple times during a live migration, a transmitted > tmem page (handle and data) will be transmitted only once, > but may be followed at some point with a handle+invalidate. > > The ordering of tmem info/pages vs current saved info/data > is flexible, but the number of pages could be very large, > so for live migrate, transmission should NOT be postponed > until the "final pass" of normal page transmission (e.g. > after the domain has been paused on the source machine). > > I also need code to verify that the destination has tmem > support and it is enabled. Only PV domains can use tmem > so no HVM changes should be necessary. > > Any pointers or suggestions welcome, especially any thoughts > on changes that might be required above libxc such as in > python code or (heaven forbid) ioemu/qemu. > > Thanks for any help! > Dan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Jun-18 10:30 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
Dan Magenheimer wrote:> Any pointers or suggestions welcome, especially any thoughts > on changes that might be required above libxc such as in > python code or (heaven forbid) ioemu/qemu. >The qemu record is saved by savevm.c:qemu_savevm_state that calls all the functions registered with register_savevm by the emulated devices. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2009-Jun-18 11:39 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
On Thu, Jun 18, 2009 at 09:51:58AM +0100, Tim Deegan wrote:> It''s still the ad-hoc stream-of-data format, I''m afraid. The whole > format needs a good kicking. It''s not even portable between 32-bit and > 64-bit tools. Gianluca (Cc''d) is just starting to look at the > save/restore code in the hopes of making it more sane, so now would be a > good time to bring up any suggestions.One possibility Anthony Liguori brought up before is using all of qemu''s container format. I''m not sure it''s ever feasible to make it fully compatible, but it certainly seems like an obvious choice to use as a base. Unfortunately, I think it''s defined-by-the-code as well. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Jun-18 12:02 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
John Levon wrote:> On Thu, Jun 18, 2009 at 09:51:58AM +0100, Tim Deegan wrote: > >> It''s still the ad-hoc stream-of-data format, I''m afraid. The whole >> format needs a good kicking. It''s not even portable between 32-bit and >> 64-bit tools. Gianluca (Cc''d) is just starting to look at the >> save/restore code in the hopes of making it more sane, so now would be a >> good time to bring up any suggestions. > > One possibility Anthony Liguori brought up before is using all of qemu''s > container format. I''m not sure it''s ever feasible to make it fully > compatible, but it certainly seems like an obvious choice to use as a > base. Unfortunately, I think it''s defined-by-the-code as well. >At least in theory is certainly feasible: just a matter or registering another savevm function for a record called "cpu" or "vcpu", that would take care of saving the guest memory using xc_domain_save. The qemu people are also maintaing save record compatibility now, so we are safe from that perspective. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2009-Jun-18 12:05 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote:> At least in theory is certainly feasible: just a matter or registering > another savevm function for a record called "cpu" or "vcpu", that would > take care of saving the guest memory using xc_domain_save.Yuck. You still need to add a bunch of metadata from xend that qemu doesn''t know about, so you''ll have to wrap the qemu output in another file format already. Wrapping the memory image inside qemu''s image is just layering for its own sake. (Also, using qemu to save a PV guests would be pretty wierd).> The qemu people are also maintaing save record compatibility now, so we > are safe from that perspective.Yes, but their code-defines-format model is rubbish. I''ve written a couple of tools that parse HVM save file info based on the Xen public headers; doing the same for a qemu-style implicit format is a PITA. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2009-Jun-18 12:26 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
On Thu, Jun 18, 2009 at 01:05:58PM +0100, Tim Deegan wrote:> At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote: > > At least in theory is certainly feasible: just a matter or registering > > another savevm function for a record called "cpu" or "vcpu", that would > > take care of saving the guest memory using xc_domain_save. > > Yuck. You still need to add a bunch of metadata from xend that qemu > doesn''t know about, so you''ll have to wrap the qemu output in another > file format already.Why? The qemu format is extensible, no? Why isn''t this just an .sxp SaveStateEntry ? Think about it from the point of view of a save-file inspector. If they can write the container-unwrapping code just once, and have specific methods for Xen-particular or kvm-particular sections, that seems like a clear win.> just layering for its own sake. (Also, using qemu to save a PV guests > would be pretty wierd).Why? We already use qemu for PV guests and that''s only going to become more common. But heck, if you like, make libxc write out qemu format.> > The qemu people are also maintaing save record compatibility now, so we > > are safe from that perspective. > > Yes, but their code-defines-format model is rubbish.Maybe now is the time to help them fix that? It''s really no worse than Xen''s code-defines-format model, headers or not. If we do need a separate container format, let''s use ELF like the core files (slightly extended as I mentioned). Just not yet another format when there''s no need. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Jun-18 12:28 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
Tim Deegan wrote:> At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote: >> At least in theory is certainly feasible: just a matter or registering >> another savevm function for a record called "cpu" or "vcpu", that would >> take care of saving the guest memory using xc_domain_save. > > Yuck. You still need to add a bunch of metadata from xend that qemu > doesn''t know about, so you''ll have to wrap the qemu output in another > file format already. Wrapping the memory image inside qemu''s image is > just layering for its own sake. (Also, using qemu to save a PV guests > would be pretty wierd).Most (but unfortunately not all) the information saved by xend are qemu command line options or easy to get from them. Certainly being backward compatible is going to be harder this way than involving xend.>> The qemu people are also maintaing save record compatibility now, so we >> are safe from that perspective. > > Yes, but their code-defines-format model is rubbish. I''ve written a > couple of tools that parse HVM save file info based on the Xen public > headers; doing the same for a qemu-style implicit format is a PITA. >We can always write an rfc that documents the format and request them to keep it up to date. We can also reuse exiting loadvm qemu code to write tools, even though it wouldn''t be as elegant as using xen public headers. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2009-Jun-18 12:43 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote:> > just layering for its own sake. (Also, using qemu to save a PV guests > > would be pretty wierd). > > Why? We already use qemu for PV guests and that''s only going to become > more common.Is it? AIUI the only thing we use qemu for in PV guests is the pvfb backend, which is just because nobody''s put in a proper library interface to that code.> > > The qemu people are also maintaing save record compatibility now, so we > > > are safe from that perspective. > > > > Yes, but their code-defines-format model is rubbish. > > Maybe now is the time to help them fix that? It''s really no worse than > Xen''s code-defines-format model, headers or not.Qemu''s code-defines-format model is actually much better than the xend/libxc code-defines-format model. :) But that''s not to say it''s the thing we should be copying.> If we do need a separate container format, let''s use ELF like the core > files (slightly extended as I mentioned). Just not yet another format > when there''s no need.ELF could work; it gives us easy access to unpacking code in several languages. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2009-Jun-18 12:47 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
On Thu, Jun 18, 2009 at 01:43:05PM +0100, Tim Deegan wrote:> At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote: > > > just layering for its own sake. (Also, using qemu to save a PV guests > > > would be pretty wierd). > > > > Why? We already use qemu for PV guests and that''s only going to become > > more common. > > Is it? AIUI the only thing we use qemu for in PV guests is the pvfb > backend, which is just because nobody''s put in a proper library > interface to that code.The console as well if xenconsoled isn''t running.> > > Yes, but their code-defines-format model is rubbish. > > > > Maybe now is the time to help them fix that? It''s really no worse than > > Xen''s code-defines-format model, headers or not. > > Qemu''s code-defines-format model is actually much better than the > xend/libxc code-defines-format model. :) But that''s not to say it''s the > thing we should be copying.Well, given we''re agreed that we should have a document of /some/ specification, why not qemu''s? Presumably we need to document the state that qemu /does/ save for HVM guests anyway, so why not the whole thing?> > If we do need a separate container format, let''s use ELF like the core > > files (slightly extended as I mentioned). Just not yet another format > > when there''s no need. > > ELF could work; it gives us easy access to unpacking code in several > languages.I admit ELF is easier to work with. It''s a shame qemu (and Xen!) didn''t use it from the start. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Jun-18 12:54 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
Tim Deegan wrote:> At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote: >>> just layering for its own sake. (Also, using qemu to save a PV guests >>> would be pretty wierd). >> Why? We already use qemu for PV guests and that''s only going to become >> more common. > > Is it? AIUI the only thing we use qemu for in PV guests is the pvfb > backend, which is just because nobody''s put in a proper library > interface to that code.We also use it for the console backend when a vfb is given to the guest.>>>> The qemu people are also maintaing save record compatibility now, so we >>>> are safe from that perspective. >>> Yes, but their code-defines-format model is rubbish. >> Maybe now is the time to help them fix that? It''s really no worse than >> Xen''s code-defines-format model, headers or not. > > Qemu''s code-defines-format model is actually much better than the > xend/libxc code-defines-format model. :) But that''s not to say it''s the > thing we should be copying.It is more about reusing than copying. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2009-Jun-18 14:18 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
At 13:47 +0100 on 18 Jun (1245332839), John Levon wrote:> Well, given we''re agreed that we should have a document of /some/ > specification, why not qemu''s? Presumably we need to document the state > that qemu /does/ save for HVM guests anyway, so why not the whole thing?If we have our own format (using ELF or indeed any other container format to break it into typed sections) then we get to use our own typespace rather than asking for a slice of theirs. :) More seriously, I don''t think it gets us enough real advantage to comensate for its disadvantages: it''s an ad-hoc, evolved, implicitly defined format, which is the sort of thing we''re trying to avoid; It encourages further aggregation of unrelated dom0 tools into qemu (I understand that some people see that as a good thing, but I don''t); and it''s tied down to the needs and history of qemu, which are not entirely aligned with Xen''s. Anyway, I think we''ve probably made our positions clear; Gianluca will presumably have his own ideas. Cheers, Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Jun-18 23:25 UTC
RE: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
> BTW, the HVM save records supplied by Xen to libxc _are_ > self-identifying and extensible (and there''s room in the header for a > version number, though by sticking to the principle of only > transferring > architectural state we''ve avoided the need to use it so far). But > they''re just dropped into the stream after the memory pages and before > xend glues on the qemu record.Semi-related, does this mean that HVM live migrate is fully-compatible between versions of Xen, e.g. either way between a Xen 3.1 machine and a Xen 3.4 machine and back again, even though new fields (TSS? EPT?) are in the save record? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2009-Jun-18 23:40 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
On 19/06/2009 00:25, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:>> BTW, the HVM save records supplied by Xen to libxc _are_ >> self-identifying and extensible (and there''s room in the header for a >> version number, though by sticking to the principle of only >> transferring >> architectural state we''ve avoided the need to use it so far). But >> they''re just dropped into the stream after the memory pages and before >> xend glues on the qemu record. > > Semi-related, does this mean that HVM live migrate > is fully-compatible between versions of Xen, e.g. > either way between a Xen 3.1 machine and a Xen 3.4 > machine and back again, even though new fields > (TSS? EPT?) are in the save record?No, that would be kind of hard since there may be new virtual hardware and other interfaces we expose in newer versions of Xen which simply do not exist in older versions. We only support migration to the same or newer versions of Xen. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Gianluca Guida
2009-Jun-22 14:28 UTC
Re: [Xen-devel] save image file format? and [RFC] tmem save/restore/migrate
Hello, xen-devel calling, of course during days off! :-) On Thu, Jun 18, 2009 at 4:18 PM, Tim Deegan<Tim.Deegan@citrix.com> wrote:> More seriously, I don''t think it gets us enough real advantage to > comensate for its disadvantages: it''s an ad-hoc, evolved, implicitly > defined format, which is the sort of thing we''re trying to avoid; It > encourages further aggregation of unrelated dom0 tools into qemu (I > understand that some people see that as a good thing, but I don''t); and > it''s tied down to the needs and history of qemu, which are not entirely > aligned with Xen''s. > > Anyway, I think we''ve probably made our positions clear; Gianluca will > presumably have his own ideas.Thanks everyone for the comments on the subject! I admit I am not actually ''passionate'' about file formats, but I do agree with those who said that using *qemu* to save a VM is ugly. I personally think that it would be yet another dangerous precedent for the idea that qemu should be used for stuff which are completely unrelated with hardware virtualization. As for the file formats, I think ELF idea is cute: modular, extendible, versioned, and easily writable by libelf. Let me know if you have further comments, and please note that I am just starting to work on a general overhaul of save/restore code, so it will take a while for me to get some code out of this talk. Thanks, Gianluca -- It was a type of people I did not know, I found them very strange and they did not inspire confidence at all. Later I learned that I had been introduced to electronic engineers. E. W. Dijkstra _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel