Xen devel - Apr 2009 - [PATCH] don''t split the whole large page while live migration with EPT

Before, when HVM guest live migration with EPT, we just write protect all the
EPT leaf entries.
Then we know the dirty pages through ept_handle_violation(), and call
p2m_change_type()
to put back the write access of the EPT entries. At that time when we use large
pages,
we used to split the large page to 4K.
It''s not necessary. Alternatively, we can just retain the large page
entries,
and put back the write access and mark dirty all the small pages in the large
page.

The patch attached removes the splitting.

Signed-off-by: Xin, Xiaohui <Xiaohui.xin@intel.com>






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 13:33 +0100 on 16 Apr (1239888814), Xin, Xiaohui
wrote:
> Before, when HVM guest live migration with EPT, we just write protect all
the EPT leaf entries.
> Then we know the dirty pages through ept_handle_violation(), and call
p2m_change_type()
> to put back the write access of the EPT entries. At that time when we use
large pages,
> we used to split the large page to 4K.
> It?s not necessary. Alternatively, we can just retain the large page
entries,
> and put back the write access and mark dirty all the small pages in the
large page.
> 
> The patch attached removes the splitting.
What effect does this have on live migration time/traffic under load?

Cheers,

Tim

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

We haven't measure the performance yet. But since we did not lock the p2m
table for p2m table reading, splitting the large page during live migration may
be dangerous while other CPU wants to read the ept entries which are being
splitted.

Thanks
Xiaohui

-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
Sent: 2009年4月16日 20:36
To: Xin, Xiaohui
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel][PATCH] don't split the whole large page while live
migration with EPT

At 13:33 +0100 on 16 Apr (1239888814), Xin, Xiaohui
wrote:
> Before, when HVM guest live migration with EPT, we just write protect all
the EPT leaf entries.
> Then we know the dirty pages through ept_handle_violation(), and call
p2m_change_type()
> to put back the write access of the EPT entries. At that time when we use
large pages,
> we used to split the large page to 4K.
> It?s not necessary. Alternatively, we can just retain the large page
entries,
> and put back the write access and mark dirty all the small pages in the
large page.
> 
> The patch attached removes the splitting.
What effect does this have on live migration time/traffic under load?

Cheers,

Tim

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 13:47 +0100 on 16 Apr (1239889668), Xin, Xiaohui
wrote:
> We haven''t measure the performance yet.
But you''re planning to? :)
> But since we did not lock the
> p2m table for p2m table reading, splitting the large page during live
> migration may be dangerous while other CPU wants to read the ept
> entries which are being splitted.
Log-dirty map reads happen with the target domain paused; all other
readers are easily made safe: build the new L1 table first and then
atomically change the l2 entry to point to it.

Cheers,

Tim.
> Thanks
> Xiaohui
> 
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
> Sent: 2009??4??16?? 20:36
> To: Xin, Xiaohui
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel][PATCH] don''t split the whole large page
while live migration with EPT
> 
> At 13:33 +0100 on 16 Apr (1239888814), Xin, Xiaohui wrote:
> > Before, when HVM guest live migration with EPT, we just write protect
all the EPT leaf entries.
> > Then we know the dirty pages through ept_handle_violation(), and call
p2m_change_type()
> > to put back the write access of the EPT entries. At that time when we
use large pages,
> > we used to split the large page to 4K.
> > It?s not necessary. Alternatively, we can just retain the large page
entries,
> > and put back the write access and mark dirty all the small pages in
the large page.
> > 
> > The patch attached removes the splitting.
> 
> What effect does this have on live migration time/traffic under load?
> 
> Cheers,
> 
> Tim
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/04/2009 13:51, "Tim Deegan" <Tim.Deegan@citrix.com> wrote:
> At 13:47 +0100 on 16 Apr (1239889668), Xin, Xiaohui wrote:
>> We haven''t measure the performance yet.
> 
> But you''re planning to? :)
This would be definitely be a prerequisite for taking this patch. Our
suspicion is that 2MB pages will be too coarse grained for many workloads,
and we''ll end up retransmitting superpages where only a handful of 4kB
sub-pages are actually dirtied.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> and we''ll end up retransmitting superpages where only a handful of
4kB
> sub-pages are actually dirtied.
Hmmm... how about enhancing live migration to handle
deltas of (2MB) pages?

If any academics are lurking about, this might make a fine research
project with good publication potential.
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Thursday, April 16, 2009 9:16 AM
> To: Tim Deegan; Xin, Xiaohui
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel][PATCH] don''t split the whole large page
while
> live migration with EPT
> 
> 
> On 16/04/2009 13:51, "Tim Deegan" <Tim.Deegan@citrix.com>
wrote:
> 
> > At 13:47 +0100 on 16 Apr (1239889668), Xin, Xiaohui wrote:
> >> We haven''t measure the performance yet.
> > 
> > But you''re planning to? :)
> 
> This would be definitely be a prerequisite for taking this patch. Our
> suspicion is that 2MB pages will be too coarse grained for 
> many workloads,
> and we''ll end up retransmitting superpages where only a handful of
4kB
> sub-pages are actually dirtied.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for the suggestions. I'd send another patch for it.

Thanks 
Xiaohui

-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
Sent: 2009年4月16日 20:52
To: Xin, Xiaohui
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel][PATCH] don't split the whole large page while live
migration with EPT

At 13:47 +0100 on 16 Apr (1239889668), Xin, Xiaohui
wrote:
> We haven't measure the performance yet.
But you're planning to? :)
> But since we did not lock the
> p2m table for p2m table reading, splitting the large page during live
> migration may be dangerous while other CPU wants to read the ept
> entries which are being splitted.
Log-dirty map reads happen with the target domain paused; all other
readers are easily made safe: build the new L1 table first and then
atomically change the l2 entry to point to it.

Cheers,

Tim.
> Thanks
> Xiaohui
> 
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
> Sent: 2009??4??16?? 20:36
> To: Xin, Xiaohui
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel][PATCH] don't split the whole large page while
live migration with EPT
> 
> At 13:33 +0100 on 16 Apr (1239888814), Xin, Xiaohui wrote:
> > Before, when HVM guest live migration with EPT, we just write protect
all the EPT leaf entries.
> > Then we know the dirty pages through ept_handle_violation(), and call
p2m_change_type()
> > to put back the write access of the EPT entries. At that time when we
use large pages,
> > we used to split the large page to 4K.
> > It?s not necessary. Alternatively, we can just retain the large page
entries,
> > and put back the write access and mark dirty all the small pages in
the large page.
> > 
> > The patch attached removes the splitting.
> 
> What effect does this have on live migration time/traffic under load?
> 
> Cheers,
> 
> Tim
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel