Keir, As you know, we try to allocate super pages in xc_hvm_build for better performance in EPT case. But the same logic is missing in xc_domain_restore. When try to add the logic, I found it is blocked by the lazy populate algorithm in restore -- only populating the pages received from source side rather than doing it at one time. So the result is the EPT guest has performance drop after live migration:( Do you have any plan to change the lazy populate algorithm? The purpose of it, I believe, is to make restore process work without guest memory layout knowledge. But different memory allocation process may cause some issues, such as perf drop after live migration. Thanks, -- best rgds, edwin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:> As you know, we try to allocate super pages in xc_hvm_build for better > performance in EPT case. But the same logic is missing in xc_domain_restore. > > When try to add the logic, I found it is blocked by the lazy populate > algorithm > in restore -- only populating the pages received from source side rather than > doing it at one time. > > So the result is the EPT guest has performance drop after live migration:( > > Do you have any plan to change the lazy populate algorithm? The purpose of it, > I > believe, is to make restore process work without guest memory layout > knowledge.Yes: If pseudo-phys page is not yet populated in target domain, AND it is first page of a 2MB extent, AND no other pages in that extent are yet populated, AND the next pages in the save-image stream populate that extent in order, THEN allocate a superpage. This is made trickier by the fact that the next 511 pages (to make the 2MB extent) might be split across a batch boundary. So we''ll have to optimistically allocate a superpage in that case, and then shatter it if it turns out that the contiguous stream of pseudo-phys addresses is broken in the next batch. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 10:03 +0100 on 26 Sep (1222423431), Keir Fraser wrote:> Yes: If pseudo-phys page is not yet populated in target domain, AND it is > first page of a 2MB extent, AND no other pages in that extent are yet > populated, AND the next pages in the save-image stream populate that extent > in order, THEN allocate a superpage. This is made trickier by the fact that > the next 511 pages (to make the 2MB extent) might be split across a batch > boundary. So we''ll have to optimistically allocate a superpage in that case, > and then shatter it if it turns out that the contiguous stream of > pseudo-phys addresses is broken in the next batch.Or flag the start-of-a-superpage frames when saving them? Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/9/08 10:08, "Tim Deegan" <Tim.Deegan@citrix.com> wrote:> At 10:03 +0100 on 26 Sep (1222423431), Keir Fraser wrote: >> Yes: If pseudo-phys page is not yet populated in target domain, AND it is >> first page of a 2MB extent, AND no other pages in that extent are yet >> populated, AND the next pages in the save-image stream populate that extent >> in order, THEN allocate a superpage. This is made trickier by the fact that >> the next 511 pages (to make the 2MB extent) might be split across a batch >> boundary. So we''ll have to optimistically allocate a superpage in that case, >> and then shatter it if it turns out that the contiguous stream of >> pseudo-phys addresses is broken in the next batch. > > Or flag the start-of-a-superpage frames when saving them?It''s not clear to me this is significantly easier to hack in on the save side, and restore side would still need modifying, and it would change the save format (albeit probably not hard to do it in a backward-compatible way). I''m not absolutely dead set on my stated approach though. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Or flag the start-of-a-superpage frames when saving them? > > It''s not clear to me this is significantly easier to hack in on thesave> side, and restore side would still need modifying, and it would change > the save format (albeit probably not hard to do it in abackward-compatible> way). I''m not absolutely dead set on my stated approach though.Also, since you typically migrate VMs to less busy machines it would be good to take the opportunity to use more 2MB frames.>> Yes: If pseudo-phys page is not yet populated in target domain, AND >> it is first page of a 2MB extent, AND no other pages in that extent >> are yet populated, AND the next pages in the save-image stream >> populate that extent in order, THEN allocate a superpage. This is >> made trickier by the fact that the next 511 pages (to make the 2MB >> extent) might be split across a batch boundary. So we''ll have to >> optimistically allocate a superpage in that case, and then shatter it>> if it turns out that the contiguous stream of pseudo-phys addressesis broken in the next batch. Rather than shattering the 2MB frame, it would be better to copy the pages out into a heap-allocated buffer, free the 2MB frame, and then allocate order-0 pages and copy the data in. This avoids needlessly fragmenting xen''s memory. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> As you know, we try to allocate super pages in xc_hvm_build for better >> performance in EPT case. But the same logic is missing in xc_domain_restore. >> >> When try to add the logic, I found it is blocked by the lazy populate >> algorithm >> in restore -- only populating the pages received from source side rather than >> doing it at one time. >> >> So the result is the EPT guest has performance drop after live migration:( >> >> Do you have any plan to change the lazy populate algorithm? The purpose of it, >> I >> believe, is to make restore process work without guest memory layout >> knowledge. >> > > Yes: If pseudo-phys page is not yet populated in target domain, AND it is > first page of a 2MB extent, AND no other pages in that extent are yet > populated, AND the next pages in the save-image stream populate that extent > in order, THEN allocate a superpage. This is made trickier by the fact that > the next 511 pages (to make the 2MB extent) might be split across a batch > boundary. So we''ll have to optimistically allocate a superpage in that case, > and then shatter it if it turns out that the contiguous stream of > pseudo-phys addresses is broken in the next batch. >It''s really tricky logic, and may make the migration process longer:( Maybe the flag the start-of-a-superpage as Tim said can make it easier. Thanks,> -- Keir > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I don't think the proposed logic is particularly tricky, and it won't be slow - the initial test of looking for the first page in a 2.MB extent acts as a good filter. It will work better than mmarking super pages on the sender. Ian ----- Original Message ----- From: xen-devel-bounces@lists.xensource.com <xen-devel-bounces@lists.xensource.com> To: Keir Fraser Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com> Sent: Sat Sep 27 10:19:15 2008 Subject: [Xen-devel] Re: super page with live migration Keir Fraser wrote:> On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> As you know, we try to allocate super pages in xc_hvm_build for better >> performance in EPT case. But the same logic is missing in xc_domain_restore. >> >> When try to add the logic, I found it is blocked by the lazy populate >> algorithm >> in restore -- only populating the pages received from source side rather than >> doing it at one time. >> >> So the result is the EPT guest has performance drop after live migration:( >> >> Do you have any plan to change the lazy populate algorithm? The purpose of it, >> I >> believe, is to make restore process work without guest memory layout >> knowledge. >> > > Yes: If pseudo-phys page is not yet populated in target domain, AND it is > first page of a 2MB extent, AND no other pages in that extent are yet > populated, AND the next pages in the save-image stream populate that extent > in order, THEN allocate a superpage. This is made trickier by the fact that > the next 511 pages (to make the 2MB extent) might be split across a batch > boundary. So we'll have to optimistically allocate a superpage in that case, > and then shatter it if it turns out that the contiguous stream of > pseudo-phys addresses is broken in the next batch. >It's really tricky logic, and may make the migration process longer:( Maybe the flag the start-of-a-superpage as Tim said can make it easier. Thanks,> -- Keir > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yes, quite apart from anything else, you¹re just punting the problem of detecting superpages (or candidates for superpages) to the sender. I¹m pretty confident it won¹t be slow and, for a live migration, the superpage logic in the restore logic is only really going to get kicked in the first batch (we could even add that as an extra condition) so won¹t extend dead time¹ at all. -- Keir On 27/9/08 11:06, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:> > > I don''t think the proposed logic is particularly tricky, and it won''t be slow > - the initial test of looking for the first page in a 2.MB extent acts as a > good filter. > > It will work better than mmarking super pages on the sender. > > Ian > > > > ----- Original Message ----- > From: xen-devel-bounces@lists.xensource.com > <xen-devel-bounces@lists.xensource.com> > To: Keir Fraser > Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com> > Sent: Sat Sep 27 10:19:15 2008 > Subject: [Xen-devel] Re: super page with live migration > > Keir Fraser wrote: >> > On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: >> > >> > >>> >> As you know, we try to allocate super pages in xc_hvm_build for better >>> >> performance in EPT case. But the same logic is missing in >>> xc_domain_restore. >>> >> >>> >> When try to add the logic, I found it is blocked by the lazy populate >>> >> algorithm >>> >> in restore -- only populating the pages received from source side rather >>> than >>> >> doing it at one time. >>> >> >>> >> So the result is the EPT guest has performance drop after live >>> migration:( >>> >> >>> >> Do you have any plan to change the lazy populate algorithm? The purpose >>> of it, >>> >> I >>> >> believe, is to make restore process work without guest memory layout >>> >> knowledge. >>> >> >> > >> > Yes: If pseudo-phys page is not yet populated in target domain, AND it is >> > first page of a 2MB extent, AND no other pages in that extent are yet >> > populated, AND the next pages in the save-image stream populate that extent >> > in order, THEN allocate a superpage. This is made trickier by the fact that >> > the next 511 pages (to make the 2MB extent) might be split across a batch >> > boundary. So we''ll have to optimistically allocate a superpage in that >> case, >> > and then shatter it if it turns out that the contiguous stream of >> > pseudo-phys addresses is broken in the next batch. >> > > > It''s really tricky logic, and may make the migration process longer:( > > Maybe the flag the start-of-a-superpage as Tim said can make it easier. > > > > Thanks, > > >> > -- Keir >> > >> > >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Ian Pratt >Sent: 2008年9月26日 18:18 > >>> if it turns out that the contiguous stream of pseudo-phys addresses >is broken in the next batch. > >Rather than shattering the 2MB frame, it would be better to copy the >pages out into a heap-allocated buffer, free the 2MB frame, and then >allocate order-0 pages and copy the data in. > >This avoids needlessly fragmenting xen''s memory. >I''m not familiar with restore proecss. Just a curious question. Is it difficult or intrusive if we still keep 2MB frame even when next 511 pages across batch boundary? Then when pages in future batches come, we just copy them into previously allocated 2MB frame covering them. I''m just not sure about the possibility for an individual 4k pseudo-phys page occuring in early patch, which then result most following superpages across batch boundary, and thus few 2MB frame can be allocated in target side. Maybe this concern is not true due to batch creation in save process? :-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just realize that superpage may extend the service shutdown time in migration process. Take a ideal convergence example for non-super page cases, just dozens of pages may keep dirty at last batch sent to target side, and thus service shutdown phase is short. However when superpage is enabled, it''s possible that those dozens of dirty pages are multiplied with a 512 factor for an extreme case where each page comes from different 2M super page. Then service shutdown phase can be longer, though not measured. Not sure how such inefficiency can be optimized... Thanks, Kevin ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser Sent: Saturday, September 27, 2008 6:49 PM To: Ian Pratt; Zhai, Edwin Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Re: super page with live migration Yes, quite apart from anything else, you''re just punting the problem of detecting superpages (or candidates for superpages) to the sender. I''m pretty confident it won''t be slow and, for a live migration, the superpage logic in the restore logic is only really going to get kicked in the first batch (we could even add that as an extra condition) so won''t extend ''dead time'' at all. -- Keir On 27/9/08 11:06, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote: I don''t think the proposed logic is particularly tricky, and it won''t be slow - the initial test of looking for the first page in a 2.MB extent acts as a good filter. It will work better than mmarking super pages on the sender. Ian ----- Original Message ----- From: xen-devel-bounces@lists.xensource.com <xen-devel-bounces@lists.xensource.com> To: Keir Fraser Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com> Sent: Sat Sep 27 10:19:15 2008 Subject: [Xen-devel] Re: super page with live migration Keir Fraser wrote:> On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> As you know, we try to allocate super pages in xc_hvm_build for better >> performance in EPT case. But the same logic is missing in xc_domain_restore. >> >> When try to add the logic, I found it is blocked by the lazy populate >> algorithm >> in restore -- only populating the pages received from source side rather than >> doing it at one time. >> >> So the result is the EPT guest has performance drop after live migration:( >> >> Do you have any plan to change the lazy populate algorithm? The purpose of it, >> I >> believe, is to make restore process work without guest memory layout >> knowledge. >> > > Yes: If pseudo-phys page is not yet populated in target domain, AND it is > first page of a 2MB extent, AND no other pages in that extent are yet > populated, AND the next pages in the save-image stream populate that extent > in order, THEN allocate a superpage. This is made trickier by the fact that > the next 511 pages (to make the 2MB extent) might be split across a batch > boundary. So we''ll have to optimistically allocate a superpage in that case, > and then shatter it if it turns out that the contiguous stream of > pseudo-phys addresses is broken in the next batch. >It''s really tricky logic, and may make the migration process longer:( Maybe the flag the start-of-a-superpage as Tim said can make it easier. Thanks,> -- Keir > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
May be best to shatter all superpages on the suspend side at start of live migration (not actually reallocate just shatter the 2MB mappings in the p2m). Needs measuring though. -- Keir On 28/9/08 08:00, "Tian, Kevin" <kevin.tian@intel.com> wrote:> Just realize that superpage may extend the service shutdown time in migration > process. Take a ideal convergence example for non-super page cases, just > dozens of pages may keep dirty at last batch sent to target side, and thus > service shutdown phase is short. However when superpage is enabled, it''s > possible that those dozens of dirty pages are multiplied with a 512 factor for > an extreme case where each page comes from different 2M super page. Then > service shutdown phase can be longer, though not measured. Not sure how such > inefficiency can be optimized... > > Thanks, > Kevin > >> >> >> >> From: xen-devel-bounces@lists.xensource.com >> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser >> Sent: Saturday, September 27, 2008 6:49 PM >> To: Ian Pratt; Zhai, Edwin >> Cc: xen-devel@lists.xensource.com >> Subject: Re: [Xen-devel] Re: super page with live migration >> >> >> Yes, quite apart from anything else, you¹re just punting the problem of >> detecting superpages (or candidates for superpages) to the sender. I¹m >> pretty confident it won¹t be slow and, for a live migration, the superpage >> logic in the restore logic is only really going to get kicked in the first >> batch (we could even add that as an extra condition) so won¹t extend dead >> time¹ at all. >> >> -- Keir >> >> On 27/9/08 11:06, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote: >> >> >>> >>> >>> I don''t think the proposed logic is particularly tricky, and it won''t be >>> slow - the initial test of looking for the first page in a 2.MB extent acts >>> as a good filter. >>> >>> It will work better than mmarking super pages on the sender. >>> >>> Ian >>> >>> >>> >>> ----- Original Message ----- >>> From: xen-devel-bounces@lists.xensource.com >>> <xen-devel-bounces@lists.xensource.com> >>> To: Keir Fraser >>> Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com> >>> Sent: Sat Sep 27 10:19:15 2008 >>> Subject: [Xen-devel] Re: super page with live migration >>> >>> Keir Fraser wrote: >>>> > On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: >>>> > >>>> > >>>>> >> As you know, we try to allocate super pages in xc_hvm_build for >>>>> better >>>>> >> performance in EPT case. But the same logic is missing in >>>>> xc_domain_restore. >>>>> >> >>>>> >> When try to add the logic, I found it is blocked by the lazy populate >>>>> >> algorithm >>>>> >> in restore -- only populating the pages received from source side >>>>> rather than >>>>> >> doing it at one time. >>>>> >> >>>>> >> So the result is the EPT guest has performance drop after live >>>>> migration:( >>>>> >> >>>>> >> Do you have any plan to change the lazy populate algorithm? The >>>>> purpose of it, >>>>> >> I >>>>> >> believe, is to make restore process work without guest memory layout >>>>> >> knowledge. >>>>> >> >>>> > >>>> > Yes: If pseudo-phys page is not yet populated in target domain, AND it >>>> is >>>> > first page of a 2MB extent, AND no other pages in that extent are yet >>>> > populated, AND the next pages in the save-image stream populate that >>>> extent >>>> > in order, THEN allocate a superpage. This is made trickier by the fact >>>> that >>>> > the next 511 pages (to make the 2MB extent) might be split across a >>>> batch >>>> > boundary. So we''ll have to optimistically allocate a superpage in that >>>> case, >>>> > and then shatter it if it turns out that the contiguous stream of >>>> > pseudo-phys addresses is broken in the next batch. >>>> > >>> >>> It''s really tricky logic, and may make the migration process longer:( >>> >>> Maybe the flag the start-of-a-superpage as Tim said can make it easier. >>> >>> >>> >>> Thanks, >>> >>> >>>> > -- Keir >>>> > >>>> > >>>> > >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Just realize that superpage may extend the service shutdown time in > migration process. Take a ideal convergence example for non-super page > cases, just dozens of pages may keep dirty at last batch sent totarget> side, and thus service shutdown phase is short. However when superpage > is enabled, it''s possible that those dozens of dirty pages are > multiplied with a 512 factor for an extreme case where each page comes > from different 2M super page. Then service shutdown phase can be > longer, though not measured. Not sure how such inefficiency can be > optimized...Yes, ideally at the start of the migrate we''d have a new shadow control op to turn off usage of 2MB pages in the shadow pagetable and HAP p2m code, and to shatter any existing ones. There''s no need to copy the memory. Ian> > Thanks, > Kevin > > > ________________________________ > > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Keir Fraser > Sent: Saturday, September 27, 2008 6:49 PM > To: Ian Pratt; Zhai, Edwin > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: super page with live migration > > > Yes, quite apart from anything else, you''re just punting the > problem of detecting superpages (or candidates for superpages) to the > sender. I''m pretty confident it won''t be slow and, for a live > migration, the superpage logic in the restore logic is only really > going to get kicked in the first batch (we could even add that as an > extra condition) so won''t extend ''dead time'' at all. > > -- Keir > > On 27/9/08 11:06, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote: > > > > > > I don''t think the proposed logic is particularly tricky, > and it won''t be slow - the initial test of looking for the first page > in a 2.MB extent acts as a good filter. > > It will work better than mmarking super pages on the > sender. > > Ian > > > > ----- Original Message ----- > From: xen-devel-bounces@lists.xensource.com <xen-devel- > bounces@lists.xensource.com> > To: Keir Fraser > Cc: xen-devel@lists.xensource.com <xen- > devel@lists.xensource.com> > Sent: Sat Sep 27 10:19:15 2008 > Subject: [Xen-devel] Re: super page with live migration > > Keir Fraser wrote: > > On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> > wrote: > > > > > >> As you know, we try to allocate super pages in > xc_hvm_build for better > >> performance in EPT case. But the same logic ismissing> in xc_domain_restore. > >> > >> When try to add the logic, I found it is blocked bythe> lazy populate > >> algorithm > >> in restore -- only populating the pages received from > source side rather than > >> doing it at one time. > >> > >> So the result is the EPT guest has performance drop > after live migration:( > >> > >> Do you have any plan to change the lazy populate > algorithm? The purpose of it, > >> I > >> believe, is to make restore process work withoutguest> memory layout > >> knowledge. > >> > > > > Yes: If pseudo-phys page is not yet populated intarget> domain, AND it is > > first page of a 2MB extent, AND no other pages in that > extent are yet > > populated, AND the next pages in the save-image stream > populate that extent > > in order, THEN allocate a superpage. This is made > trickier by the fact that > > the next 511 pages (to make the 2MB extent) might be > split across a batch > > boundary. So we''ll have to optimistically allocate a > superpage in that case, > > and then shatter it if it turns out that thecontiguous> stream of > > pseudo-phys addresses is broken in the next batch. > > > > It''s really tricky logic, and may make the migration > process longer:( > > Maybe the flag the start-of-a-superpage as Tim said can > make it easier. > > > > Thanks, > > > > -- Keir > > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m not familiar with restore proecss. Just a curious question. > Is it difficult or intrusive if we still keep 2MB frame even when > next 511 pages across batch boundary?We should optimistically allocate a 2MB frame if the rest of the batch we can see is contiguous.> Then when pages in > future batches come, we just copy them into previously > allocated 2MB frame covering them.Yes, they''ll already have p2m table entries as a result of the earlier allocation. If the frames turn out not to be contiguous, we should allocate a buffer in the heap, copy the earlier pages into it, free the 2MB frame, allocate 4KB frames and copy the data in.> I''m just not sure about > the possibility for an individual 4k pseudo-phys page occuring > in early patch, which then result most following superpages > across batch boundary, and thus few 2MB frame can be > allocated in target side. Maybe this concern is not true due to > batch creation in save process? :-)The first iteration sends frames in-order, so it should be fine. Ian> Thanks, > Kevin_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yes, that makes sense. We can do experiment and comparison later. Thanks, Kevin ________________________________ From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] Sent: Sunday, September 28, 2008 5:31 PM To: Tian, Kevin; Ian Pratt; Zhai, Edwin Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Re: super page with live migration May be best to shatter all superpages on the suspend side at start of live migration (not actually reallocate - just shatter the 2MB mappings in the p2m). Needs measuring though. -- Keir On 28/9/08 08:00, "Tian, Kevin" <kevin.tian@intel.com> wrote: Just realize that superpage may extend the service shutdown time in migration process. Take a ideal convergence example for non-super page cases, just dozens of pages may keep dirty at last batch sent to target side, and thus service shutdown phase is short. However when superpage is enabled, it''s possible that those dozens of dirty pages are multiplied with a 512 factor for an extreme case where each page comes from different 2M super page. Then service shutdown phase can be longer, though not measured. Not sure how such inefficiency can be optimized... Thanks, Kevin ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser Sent: Saturday, September 27, 2008 6:49 PM To: Ian Pratt; Zhai, Edwin Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Re: super page with live migration Yes, quite apart from anything else, you''re just punting the problem of detecting superpages (or candidates for superpages) to the sender. I''m pretty confident it won''t be slow and, for a live migration, the superpage logic in the restore logic is only really going to get kicked in the first batch (we could even add that as an extra condition) so won''t extend ''dead time'' at all. -- Keir On 27/9/08 11:06, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote: I don''t think the proposed logic is particularly tricky, and it won''t be slow - the initial test of looking for the first page in a 2.MB extent acts as a good filter. It will work better than mmarking super pages on the sender. Ian ----- Original Message ----- From: xen-devel-bounces@lists.xensource.com <xen-devel-bounces@lists.xensource.com> To: Keir Fraser Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com> Sent: Sat Sep 27 10:19:15 2008 Subject: [Xen-devel] Re: super page with live migration Keir Fraser wrote:> On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> As you know, we try to allocate super pages in xc_hvm_build for better >> performance in EPT case. But the same logic is missing in xc_domain_restore. >> >> When try to add the logic, I found it is blocked by the lazy populate >> algorithm >> in restore -- only populating the pages received from source side rather than >> doing it at one time. >> >> So the result is the EPT guest has performance drop after live migration:( >> >> Do you have any plan to change the lazy populate algorithm? The purpose of it, >> I >> believe, is to make restore process work without guest memory layout >> knowledge. >> > > Yes: If pseudo-phys page is not yet populated in target domain, AND it is > first page of a 2MB extent, AND no other pages in that extent are yet > populated, AND the next pages in the save-image stream populate that extent > in order, THEN allocate a superpage. This is made trickier by the fact that > the next 511 pages (to make the 2MB extent) might be split across a batch > boundary. So we''ll have to optimistically allocate a superpage in that case, > and then shatter it if it turns out that the contiguous stream of > pseudo-phys addresses is broken in the next batch. >It''s really tricky logic, and may make the migration process longer:( Maybe the flag the start-of-a-superpage as Tim said can make it easier. Thanks,> -- Keir > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Ian Pratt [mailto:Ian.Pratt@eu.citrix.com] >Sent: Sunday, September 28, 2008 10:17 PM > >> I''m not familiar with restore proecss. Just a curious question. >> Is it difficult or intrusive if we still keep 2MB frame even when >> next 511 pages across batch boundary? > >We should optimistically allocate a 2MB frame if the rest of the batch >we can see is contiguous. > >> Then when pages in >> future batches come, we just copy them into previously >> allocated 2MB frame covering them. > >Yes, they''ll already have p2m table entries as a result of the earlier >allocation. > >If the frames turn out not to be contiguous, we should >allocate a buffer >in the heap, copy the earlier pages into it, free the 2MB frame, >allocate 4KB frames and copy the data in. > >> I''m just not sure about >> the possibility for an individual 4k pseudo-phys page occuring >> in early patch, which then result most following superpages >> across batch boundary, and thus few 2MB frame can be >> allocated in target side. Maybe this concern is not true due to >> batch creation in save process? :-) > >The first iteration sends frames in-order, so it should be fine. >OK, same page then. I misunderstood your comments a bit before. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel