Shriram Rajagopalan
2011-Jun-23 01:04 UTC
[Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
# HG changeset patch # User Shriram Rajagopalan <rshriram@cs.ubc.ca> # Date 1308790913 25200 # Node ID 8bef913a3c4d14d2246086ef30a8e80f45ad1beb # Parent 9eed27800ff6a2e6d73f138f20af072c1b41925e tools/libxc: Document checkpoint compression in xg_save_restore.h Add comments to xg_save_restore.h explaining changes in Remus wire protocol when checkpoint compression is enabled. Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca> diff -r 9eed27800ff6 -r 8bef913a3c4d tools/libxc/xg_save_restore.h --- a/tools/libxc/xg_save_restore.h Wed Jun 22 06:34:55 2011 -0700 +++ b/tools/libxc/xg_save_restore.h Wed Jun 22 18:01:53 2011 -0700 @@ -82,9 +82,24 @@ * page data : PAGE_SIZE bytes for each page marked present in PFN * array * + * In case of Remus with checkpoint compression, since the compressed page data + * can be of variable size, only the pfn array is sent with a +ve chunk type. + * * If the chunk type is -ve then chunk consists of one of a number of * metadata types. See definitions of XC_SAVE_ID_* below. * + * If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the + * chunk consists of compressed page data, in the following format: + * + * unsigned long : Size of the compressed chunk to follow + * compressed page data : variable length data of size indicated above. + * This chunk consists of compressed page data. The + * number of pages in one chunk varies with respect + * to amount of space available in the sender''s + * output buffer. + * + * There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA. + * * If chunk type is 0 then body phase is complete. * * TAIL PHASE _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Jun-23 14:56 UTC
Re: [Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Thu, 2011-06-23 at 02:04 +0100, Shriram Rajagopalan wrote:> # HG changeset patch > # User Shriram Rajagopalan <rshriram@cs.ubc.ca> > # Date 1308790913 25200 > # Node ID 8bef913a3c4d14d2246086ef30a8e80f45ad1beb > # Parent 9eed27800ff6a2e6d73f138f20af072c1b41925e > tools/libxc: Document checkpoint compression in xg_save_restore.h > > Add comments to xg_save_restore.h explaining changes in Remus > wire protocol when checkpoint compression is enabled.Thanks!> Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca> > > diff -r 9eed27800ff6 -r 8bef913a3c4d tools/libxc/xg_save_restore.h > --- a/tools/libxc/xg_save_restore.h Wed Jun 22 06:34:55 2011 -0700 > +++ b/tools/libxc/xg_save_restore.h Wed Jun 22 18:01:53 2011 -0700 > @@ -82,9 +82,24 @@ > * page data : PAGE_SIZE bytes for each page marked present in PFN > * array > * > + * In case of Remus with checkpoint compression, since the compressed page data > + * can be of variable size, only the pfn array is sent with a +ve chunk type. > + *Is this true on every round or only for second and subsequent rounds? How do we know on the receiving end whether or not we are using checkpoint compression? Is there an XC_SAVE_ID which is sent to trigger this mode?> * If the chunk type is -ve then chunk consists of one of a number of > * metadata types. See definitions of XC_SAVE_ID_* below. > * > + * If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the > + * chunk consists of compressed page data, in the following format: > + * > + * unsigned long : Size of the compressed chunk to follow > + * compressed page data : variable length data of size indicated above. > + * This chunk consists of compressed page data. The > + * number of pages in one chunk varies with respect > + * to amount of space available in the sender''s > + * output buffer. > + * > + * There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA.Having seen an XC_SAVE_ID_COMPRESSED_DATA can you ever see a +ve chunk type at that point? IOW is XC_SAVE_ID_COMPRESSED_DATA also the trigger I mentioned above? Part of me thinks this bit belongs below with the #define and the other part thinks its too big to fit in nicely there so it is better here. Ian.> * If chunk type is 0 then body phase is complete. > * > * TAIL PHASE > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Shriram Rajagopalan
2011-Jun-23 15:16 UTC
Re: [Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Thu, Jun 23, 2011 at 10:56 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote:> On Thu, 2011-06-23 at 02:04 +0100, Shriram Rajagopalan wrote: > > # HG changeset patch > > # User Shriram Rajagopalan <rshriram@cs.ubc.ca> > > # Date 1308790913 25200 > > # Node ID 8bef913a3c4d14d2246086ef30a8e80f45ad1beb > > # Parent 9eed27800ff6a2e6d73f138f20af072c1b41925e > > tools/libxc: Document checkpoint compression in xg_save_restore.h > > > > Add comments to xg_save_restore.h explaining changes in Remus > > wire protocol when checkpoint compression is enabled. > > Thanks! > > > Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca> > > > > diff -r 9eed27800ff6 -r 8bef913a3c4d tools/libxc/xg_save_restore.h > > --- a/tools/libxc/xg_save_restore.h Wed Jun 22 06:34:55 2011 -0700 > > +++ b/tools/libxc/xg_save_restore.h Wed Jun 22 18:01:53 2011 -0700 > > @@ -82,9 +82,24 @@ > > * page data : PAGE_SIZE bytes for each page marked present > in PFN > > * array > > * > > + * In case of Remus with checkpoint compression, since the compressed > page data > > + * can be of variable size, only the pfn array is sent with a +ve chunk > type. > > + * > > Is this true on every round or only for second and subsequent rounds? >How do we know on the receiving end whether or not we are using> checkpoint compression? Is there an XC_SAVE_ID which is sent to trigger > this mode? > > Actually, after the last iteration, the primary sends aXC_SAVE_ID_ENABLE_COMPRESSION and then for further rounds, the receiver stops expecting page data following the pfn array. Instead it waits for XC_SAVE_ID_COMPRESSED_DATA. Thanks for pointing it out. I ll document it.> > * If the chunk type is -ve then chunk consists of one of a number of > > * metadata types. See definitions of XC_SAVE_ID_* below. > > * > > + * If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then > the > > + * chunk consists of compressed page data, in the following format: > > + * > > + * unsigned long : Size of the compressed chunk to follow > > + * compressed page data : variable length data of size indicated > above. > > + * This chunk consists of compressed page > data. The > > + * number of pages in one chunk varies with > respect > > + * to amount of space available in the > sender''s > > + * output buffer. > > + * > > + * There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA. > > Having seen an XC_SAVE_ID_COMPRESSED_DATA can you ever see a +ve chunk > type at that point? IOW is XC_SAVE_ID_COMPRESSED_DATA also the trigger I > mentioned above? > > Answered above. Its a separate trigger. That said, if you meant "can Iexpect a +ve chunk ''after'' a XC_SAVE_ID_COMPRESSED_DATA", yes that is possible. It happens when there are too many dirty pages to fit in the sender''s compression buffer. Sender basically blocks, sends out the compressed chunk and moves on to the next batch of pages. This is a corner case.> Part of me thinks this bit belongs below with the #define and the other > part thinks its too big to fit in nicely there so it is better here. > > Ian. > > > * If chunk type is 0 then body phase is complete. > > * > > * TAIL PHASE > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Jun-24 08:54 UTC
Re: [Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Thu, 2011-06-23 at 16:16 +0100, Shriram Rajagopalan wrote:> Actually, after the last iteration, the primary sends a > XC_SAVE_ID_ENABLE_COMPRESSION and then for further rounds, the > receiver stops expecting page data following the pfn array. Instead it > waits for XC_SAVE_ID_COMPRESSED_DATA. Thanks for pointing it out. I ll > document it.[...]> Answered above. Its a separate trigger. That said, if you meant "can I > expect a +ve chunk ''after'' a XC_SAVE_ID_COMPRESSED_DATA", yes that is > possible. It happens when there are too many dirty pages to fit in the > sender''s compression buffer. Sender basically blocks, sends out the > compressed chunk and moves on to the next batch of pages. This is a > corner case.I think I''m misunderstanding. When there are too many dirty pages you send out a standard +ve chunk, including the page data? (presumably in order to catch up). If so then how does this mesh with the statement that once you''ve seen an XC_SAVE_ID_COMPRESSED_DATA you don''t expect page data in a +ve chunk anymore? Back in the original patch:> + * compressed page data : variable length data of size indicated above. > + * This chunk consists of compressed page data. The > + * number of pages in one chunk varies with respect > + * to amount of space available in the sender''s > + * output buffer.What''s the format of this compressed page data? So is the sequence: +16 (e.g.) +ve chunk unsigned long[16] PFN array NOT page-data (because we''ve seen XC_SAVE_ID_COMPRESSED_DATA) XC_SAVE_ID_COMPRESSED_DATA TAG N Length of compressed data batch#1 N bytes of DATA, batch #1 Decompresses to e.g. 7 pages XC SAVE_ID_COMPRESSED_DATA TAG M Length of compressed data batch#2 M bytes of DATA, batch #2 Decompresses to e.g. 9 pages So now we have the originally specified 16 pages. Do we guarantee that we will always see enough instances of XC_SAVE_ID_COMPRESSED_DATA to total to the +ve chunk specified number of pages? Are they always contiguous? How does the sequence of events differ in the corner case of too many dirty pages? Do we abort e.g. before the second XC_SAVE_ID_COMPRESSED_DATA and go back to the +ve chunk stage? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Shriram Rajagopalan
2011-Jun-24 13:49 UTC
Re: [Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Fri, Jun 24, 2011 at 4:54 AM, Ian Campbell <Ian.Campbell@eu.citrix.com>wrote:> On Thu, 2011-06-23 at 16:16 +0100, Shriram Rajagopalan wrote: > > > > Actually, after the last iteration, the primary sends a > > XC_SAVE_ID_ENABLE_COMPRESSION and then for further rounds, the > > receiver stops expecting page data following the pfn array. Instead it > > waits for XC_SAVE_ID_COMPRESSED_DATA. Thanks for pointing it out. I ll > > document it. > [...] > > > Answered above. Its a separate trigger. That said, if you meant "can I > > expect a +ve chunk ''after'' a XC_SAVE_ID_COMPRESSED_DATA", yes that is > > possible. It happens when there are too many dirty pages to fit in the > > sender''s compression buffer. Sender basically blocks, sends out the > > compressed chunk and moves on to the next batch of pages. This is a > > corner case. > > I think I''m misunderstanding. When there are too many dirty pages you > send out a standard +ve chunk, including the page data? (presumably in > order to catch up). If so then how does this mesh with the statement > that once you''ve seen an XC_SAVE_ID_COMPRESSED_DATA you don''t expect > page data in a +ve chunk anymore? > > That was a good catch. I thought I ll keep the corner cases out ofxg_save_restore.h but from the way you pointed out, I think I ll document everything. Answers below.> Back in the original patch: > > + * compressed page data : variable length data of size indicated > above. > > + * This chunk consists of compressed page > data. The > > + * number of pages in one chunk varies with > respect > > + * to amount of space available in the > sender''s > > + * output buffer. > > What''s the format of this compressed page data? > > compressed_data = (BEGIN_PAGE,<deltas>*) | (FULL_PAGE,4096 bytes)delta = <+ve offset in page (2byte), value (4byte)> BEGIN_PAGE = is a dummy delta with offset = -100 and value = 0 FULL_PAGE = is a dummy delta with offset = -101 and value = 0> So is the sequence: > +16 (e.g.) +ve chunk > unsigned long[16] PFN array > NOT page-data (because we''ve seen XC_SAVE_ID_COMPRESSED_DATA) >XC_SAVE_ID_COMPRESSED_DATA TAG> N Length of compressed data batch#1 >N bytes of DATA, batch #1 Decompresses to e.g. 7 pages> XC SAVE_ID_COMPRESSED_DATA TAG > M Length of compressed data batch#2 > M bytes of DATA, batch #2 Decompresses to e.g. 9 pages > >So now we have the originally specified 16 pages. The sequence goes like this. +16 +ve chunk unsigned long[16] PFN array NO page-data (because we''ve seen XC_SAVE_ID_ENABLE_COMPRESSION) +100 +ve chunk unsigned long[100] PFN array +50 (e.g.) +ve chunk unsigned long[50] PFN array XC_SAVE_ID_COMPRESSED_DATA TAG N Length of compressed data (no batches. explained below) N bytes of DATA Decompresses to e.g. 166 pages XC_SAVE_ID_* other xc save tags.. tailbuf Do we guarantee that> we will always see enough instances of XC_SAVE_ID_COMPRESSED_DATA to > total to the +ve chunk specified number of pages?Yes. If the number of dirty pages < the sender''s page buffer size in xc_remus.c (not to be confused with pagebuf structure in xc_domain_restore), then there will be only one instance of XC_SAVE_ID_COMPRESSED_DATA, holding all the compressed pages. The page buffer can hold upto 8196 pages. So, most of the time, there will be only one XC_SAVE_ID_COMPRESSED_DATA, followed by the compressed chunk> Are they always > contiguous? > > The order of compressed pages arriving, is same as the order in which theywould have been sent if compression logic was not present.> How does the sequence of events differ in the corner case of too many > dirty pages? Do we abort e.g. before the second > XC_SAVE_ID_COMPRESSED_DATA and go back to the +ve chunk stage? > > In case there are too many dirty pages, there would be aXC_SAVE_ID_COMPRESSED_DATA in the midst of an otherwise contiguous series of +ve chunks. For e.g. say if there are 9000 dirty pages, all of which are valid. +1024 +ve chunk (batch #1) unsigned long[1024] PFN array +1024 +ve chunk (batch #2) unsigned long[1024] PFN array +1024 +ve chunk (batch #3) unsigned long[1024] PFN array +1024 +ve chunk (batch #4) unsigned long[1024] PFN array +1024 +ve chunk (batch #5) unsigned long[1024] PFN array +1024 +ve chunk (batch #6) unsigned long[1024] PFN array +1024 +ve chunk (batch #7) unsigned long[1024] PFN array +1024 +ve chunk (batch #8) unsigned long[1024] PFN array XC_SAVE_ID_COMPRESSED_DATA TAG N #length of compressed data N bytes of DATA (decompresses to 8196 pages) +4 +ve chunk unsigned long[5] PFN array XC_SAVE_ID_COMPRESSED_DATA TAG M #length of compressed data M bytes of DATA (decompresses to 4 pages) XC_SAVE_ID_* other xc save tags.. tailbuf shriram Ian.> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Jun-24 14:43 UTC
Re: [Xen-devel] [PATCH] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Fri, 2011-06-24 at 14:49 +0100, Shriram Rajagopalan wrote:> On Fri, Jun 24, 2011 at 4:54 AM, Ian Campbell > <Ian.Campbell@eu.citrix.com> wrote: > On Thu, 2011-06-23 at 16:16 +0100, Shriram Rajagopalan wrote: > > > > Actually, after the last iteration, the primary sends a > > XC_SAVE_ID_ENABLE_COMPRESSION and then for further rounds, > the > > receiver stops expecting page data following the pfn array. > Instead it > > waits for XC_SAVE_ID_COMPRESSED_DATA. Thanks for pointing it > out. I ll > > document it. > > [...] > > > Answered above. Its a separate trigger. That said, if you > meant "can I > > expect a +ve chunk ''after'' a XC_SAVE_ID_COMPRESSED_DATA", > yes that is > > possible. It happens when there are too many dirty pages to > fit in the > > sender''s compression buffer. Sender basically blocks, sends > out the > > compressed chunk and moves on to the next batch of pages. > This is a > > corner case. > > > I think I''m misunderstanding. When there are too many dirty > pages you > send out a standard +ve chunk, including the page data? > (presumably in > order to catch up). If so then how does this mesh with the > statement > that once you''ve seen an XC_SAVE_ID_COMPRESSED_DATA you don''t > expect > page data in a +ve chunk anymore? > > That was a good catch. I thought I ll keep the corner cases out of > xg_save_restore.h > but from the way you pointed out, I think I ll document everything. > Answers below.Thanks, I think the weird corner cases are actually the most important thing to document...> The sequence goes like this.[...]> > In case there are too many dirty pages, there would be a > XC_SAVE_ID_COMPRESSED_DATA in > the midst of an otherwise contiguous series of +ve chunks. For e.g. > say if there are 9000 dirty pages, > all of which are valid.Thanks, I think I get it. It might be worth making it clear in the docs that +ve and XC_SAVE_ID_COMPRESSED_DATA can effectively be mixed/interleaved arbitrarily but that the receiver will always have seen more +ve page array entires than pages in compressed form. i.e. that the amount of decompressed pages received can never exceed where page array the +ve chunks have reached. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Shriram Rajagopalan
2011-Jun-25 20:50 UTC
[Xen-devel] [PATCH v2] tools/libxc: Document checkpoint compression in xg_save_restore.h
# HG changeset patch # User Shriram Rajagopalan <rshriram@cs.ubc.ca> # Date 1309034380 25200 # Node ID 1547d25fe51b0c84b908ca4fce02568f77b431b0 # Parent c31e9249893d309655a8e739ba2ecb07d2c0148b tools/libxc: Document checkpoint compression in xg_save_restore.h Add comments to xg_save_restore.h explaining changes in Remus wire protocol when checkpoint compression is enabled. Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca> diff -r c31e9249893d -r 1547d25fe51b tools/libxc/xg_save_restore.h --- a/tools/libxc/xg_save_restore.h Sat Jun 18 20:52:07 2011 -0700 +++ b/tools/libxc/xg_save_restore.h Sat Jun 25 13:39:40 2011 -0700 @@ -67,7 +67,7 @@ * * consists of p2m_size bytes comprising an array of xen_pfn_t sized entries. * - * BODY PHASE + * BODY PHASE - Format A (for live migration or Remus without compression) * ---------- * * A series of chunks with a common header: @@ -87,6 +87,113 @@ * * If chunk type is 0 then body phase is complete. * + * + * BODY PHASE - Format B (for Remus with compression) + * ---------- + * + * A series of chunks with a common header: + * int : chunk type + * + * If the chunk type is +ve then chunk contains array of PFNs corresponding + * to guest memory and type contains the number of PFNs in the batch: + * + * unsigned long[] : PFN array, length == number of pages in batch + * Each entry consists of XEN_DOMCTL_PFINFO_* + * in bits 31-28 and the PFN number in bits 27-0. + * + * If the chunk type is -ve then chunk consists of one of a number of + * metadata types. See definitions of XC_SAVE_ID_* below. + * + * If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the + * chunk consists of compressed page data, in the following format: + * + * unsigned long : Size of the compressed chunk to follow + * compressed data : variable length data of size indicated above. + * This chunk consists of compressed page data. + * The number of pages in one chunk depends on + * the amount of space available in the sender''s + * output buffer. + * + * Format of compressed data: + * compressed_data = (BEGIN_PAGE,<deltas>*) | (FULL_PAGE,4096 bytes) + * delta = <+ve offset in page (2byte), value (4byte)> + * BEGIN_PAGE = a dummy delta with offset = -100 and value = 0 + * FULL_PAGE = a dummy delta with offset = -101 and value = 0 + * + * If chunk type is 0 then body phase is complete. + * + * There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA, + * containing compressed pages. The compressed chunks are collated to form + * one single compressed chunk for the entire iteration. The number of pages + * present in this final compressed chunk will be equal to the total number + * of valid PFNs specified by the +ve chunks. + * + * At the sender side, compressed pages are inserted into the output stream + * in the same order as they would have been if compression logic was absent. + * + * Until last iteration, the BODY is sent in Format A, to maintain live + * migration compatibility with receivers of older Xen versions. + * At the last iteration, if Remus compression was enabled, the sender sends + * a trigger, XC_SAVE_ID_ENABLE_COMPRESSION to tell the receiver to parse the + * BODY in Format B from the next iteration onwards. + * + * An example sequence of chunks received in Format B: + * +16 +ve chunk + * unsigned long[16] PFN array + * +100 +ve chunk + * unsigned long[100] PFN array + * +50 +ve chunk + * unsigned long[50] PFN array + * + * XC_SAVE_ID_COMPRESSED_DATA TAG + * N Length of compressed data + * N bytes of DATA Decompresses to 166 pages + * + * XC_SAVE_ID_* other xc save chunks + * 0 END BODY TAG + * + * Corner case with checkpoint compression: + * At sender side, after pausing the domain, dirty pages are usually + * copied out to a temporary buffer. After the domain is resumed, + * compression is done and the compressed chunk(s) are sent, followed by + * other XC_SAVE_ID_* chunks. + * If the temporary buffer gets full while scanning for dirty pages, + * the sender stops buffering of dirty pages, compresses the temporary + * buffer and sends the compressed data with XC_SAVE_ID_COMPRESSED_DATA. + * The sender then resumes the buffering of dirty pages and continues + * scanning for the dirty pages. + * For e.g., assume that the temporary buffer can hold 4096 pages and + * there are 5000 dirty pages. The following is the sequence of chunks + * that the receiver will see: + * + * +1024 +ve chunk + * unsigned long[1024] PFN array + * +1024 +ve chunk + * unsigned long[1024] PFN array + * +1024 +ve chunk + * unsigned long[1024] PFN array + * +1024 +ve chunk + * unsigned long[1024] PFN array + * + * XC_SAVE_ID_COMPRESSED_DATA TAG + * N Length of compressed data + * N bytes of DATA Decompresses to 4096 pages + * + * +4 +ve chunk + * unsigned long[4] PFN array + * + * XC_SAVE_ID_COMPRESSED_DATA TAG + * M Length of compressed data + * M bytes of DATA Decompresses to 4 pages + * + * XC_SAVE_ID_* other xc save chunks + * 0 END BODY TAG + * + * In other words, XC_SAVE_ID_COMPRESSED_DATA can be interleaved with + * +ve chunks arbitrarily. But at the receiver end, the following condition + * always holds true until the end of BODY PHASE: + * num(PFN entries +ve chunks) >= num(pages received in compressed form) + * * TAIL PHASE * ---------- * _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Jun-27 08:54 UTC
[Xen-devel] Re: [PATCH v2] tools/libxc: Document checkpoint compression in xg_save_restore.h
On Sat, 2011-06-25 at 21:50 +0100, Shriram Rajagopalan wrote:> # HG changeset patch > # User Shriram Rajagopalan <rshriram@cs.ubc.ca> > # Date 1309034380 25200 > # Node ID 1547d25fe51b0c84b908ca4fce02568f77b431b0 > # Parent c31e9249893d309655a8e739ba2ecb07d2c0148b > tools/libxc: Document checkpoint compression in xg_save_restore.h > > Add comments to xg_save_restore.h explaining changes in Remus > wire protocol when checkpoint compression is enabled. > > Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>Thanks Shriram. Acked-by: Ian Campbell <ian.campbell@citrix.com>> > diff -r c31e9249893d -r 1547d25fe51b tools/libxc/xg_save_restore.h > --- a/tools/libxc/xg_save_restore.h Sat Jun 18 20:52:07 2011 -0700 > +++ b/tools/libxc/xg_save_restore.h Sat Jun 25 13:39:40 2011 -0700 > @@ -67,7 +67,7 @@ > * > * consists of p2m_size bytes comprising an array of xen_pfn_t sized entries. > * > - * BODY PHASE > + * BODY PHASE - Format A (for live migration or Remus without compression) > * ---------- > * > * A series of chunks with a common header: > @@ -87,6 +87,113 @@ > * > * If chunk type is 0 then body phase is complete. > * > + * > + * BODY PHASE - Format B (for Remus with compression) > + * ---------- > + * > + * A series of chunks with a common header: > + * int : chunk type > + * > + * If the chunk type is +ve then chunk contains array of PFNs corresponding > + * to guest memory and type contains the number of PFNs in the batch: > + * > + * unsigned long[] : PFN array, length == number of pages in batch > + * Each entry consists of XEN_DOMCTL_PFINFO_* > + * in bits 31-28 and the PFN number in bits 27-0. > + * > + * If the chunk type is -ve then chunk consists of one of a number of > + * metadata types. See definitions of XC_SAVE_ID_* below. > + * > + * If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the > + * chunk consists of compressed page data, in the following format: > + * > + * unsigned long : Size of the compressed chunk to follow > + * compressed data : variable length data of size indicated above. > + * This chunk consists of compressed page data. > + * The number of pages in one chunk depends on > + * the amount of space available in the sender''s > + * output buffer. > + * > + * Format of compressed data: > + * compressed_data = (BEGIN_PAGE,<deltas>*) | (FULL_PAGE,4096 bytes) > + * delta = <+ve offset in page (2byte), value (4byte)> > + * BEGIN_PAGE = a dummy delta with offset = -100 and value = 0 > + * FULL_PAGE = a dummy delta with offset = -101 and value = 0 > + * > + * If chunk type is 0 then body phase is complete. > + * > + * There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA, > + * containing compressed pages. The compressed chunks are collated to form > + * one single compressed chunk for the entire iteration. The number of pages > + * present in this final compressed chunk will be equal to the total number > + * of valid PFNs specified by the +ve chunks. > + * > + * At the sender side, compressed pages are inserted into the output stream > + * in the same order as they would have been if compression logic was absent. > + * > + * Until last iteration, the BODY is sent in Format A, to maintain live > + * migration compatibility with receivers of older Xen versions. > + * At the last iteration, if Remus compression was enabled, the sender sends > + * a trigger, XC_SAVE_ID_ENABLE_COMPRESSION to tell the receiver to parse the > + * BODY in Format B from the next iteration onwards. > + * > + * An example sequence of chunks received in Format B: > + * +16 +ve chunk > + * unsigned long[16] PFN array > + * +100 +ve chunk > + * unsigned long[100] PFN array > + * +50 +ve chunk > + * unsigned long[50] PFN array > + * > + * XC_SAVE_ID_COMPRESSED_DATA TAG > + * N Length of compressed data > + * N bytes of DATA Decompresses to 166 pages > + * > + * XC_SAVE_ID_* other xc save chunks > + * 0 END BODY TAG > + * > + * Corner case with checkpoint compression: > + * At sender side, after pausing the domain, dirty pages are usually > + * copied out to a temporary buffer. After the domain is resumed, > + * compression is done and the compressed chunk(s) are sent, followed by > + * other XC_SAVE_ID_* chunks. > + * If the temporary buffer gets full while scanning for dirty pages, > + * the sender stops buffering of dirty pages, compresses the temporary > + * buffer and sends the compressed data with XC_SAVE_ID_COMPRESSED_DATA. > + * The sender then resumes the buffering of dirty pages and continues > + * scanning for the dirty pages. > + * For e.g., assume that the temporary buffer can hold 4096 pages and > + * there are 5000 dirty pages. The following is the sequence of chunks > + * that the receiver will see: > + * > + * +1024 +ve chunk > + * unsigned long[1024] PFN array > + * +1024 +ve chunk > + * unsigned long[1024] PFN array > + * +1024 +ve chunk > + * unsigned long[1024] PFN array > + * +1024 +ve chunk > + * unsigned long[1024] PFN array > + * > + * XC_SAVE_ID_COMPRESSED_DATA TAG > + * N Length of compressed data > + * N bytes of DATA Decompresses to 4096 pages > + * > + * +4 +ve chunk > + * unsigned long[4] PFN array > + * > + * XC_SAVE_ID_COMPRESSED_DATA TAG > + * M Length of compressed data > + * M bytes of DATA Decompresses to 4 pages > + * > + * XC_SAVE_ID_* other xc save chunks > + * 0 END BODY TAG > + * > + * In other words, XC_SAVE_ID_COMPRESSED_DATA can be interleaved with > + * +ve chunks arbitrarily. But at the receiver end, the following condition > + * always holds true until the end of BODY PHASE: > + * num(PFN entries +ve chunks) >= num(pages received in compressed form) > + * > * TAIL PHASE > * ---------- > *_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel