Xen devel - Jan 2008 - [PATCH]Provide 3 times continously writes check and unshadow the guest page

The patch checks if the guest continuously writes 3 times to the same
guest page table pages, if yes, then unshadow the guest pages. The idea
is copied from the KVM side. And since the idea has overlapped with the
now optimization in the now Xen shadow code, so the patch removes the
optimization part which when the guest writes 2 continuously 0 to the
same page, then unshadow the guest page.

 

In the latest Xen, if we use iperf to test the NIC assigned to HVM
guest, we will find that the performance data is very unstable and low
especially with an ia32pae guest. The root cause is that Xen holds the
guest page table pages even when they changes to data pages, and at that
time if guest write pure data to the paged, Xen will get many
unnecessary pages faults. It hurts the iperf performance a lot. The
patch resolves the issue (compare to the old optimization). And
considering the side effects to the other benchmark, we have a test
result as followed.

 

Benchmark

Guest OS

with out the patch

with the patch

Gain(+)/Downgrade(-)

 

 

 

 

 

Sysbench

Windows XP (32pae)

58.4704

58.8196

-0.59%

 

Windows XP (ia32e)

31.2266

31.6322

-1.30%

Specjbb

Linux (32pae)

1530.52

1525.27

-0.34%

 

Linux (ia32e

975.94

970.44

-0.56%

Kernel Build

Linux (32pae)

1m44s

1m43s

+0.96%

 

Linux (ia32e)

2m38s

2m23s

+9.5%

Sysmark2007

 

 

 

 

   E-learning

Windows XP (32pae)

55

56

+1.8%

   Video Creation

Windows XP (32pae)

49

48

-2%

   Productivity (cannot run)

 

 

 

 

   3D (cannot run)

 

 

 

 

Iperf

Linux (32pae)

Very unstable from ~50M/bs to 500M/bs

Stable ~550M/bs

 

 

 

Signed-off-by Xin Xiaohui xiaohui.xin@intel.com

Signed-off-by Dong Eddie <eddie.dong@intel.com>

 

 

 

 

 

 

 




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

At 13:41 +0800 on 23 Jan (1201095692), Xin, Xiaohui
wrote:
> The patch checks if the guest continuously writes 3 times to the same
> guest page table pages, if yes, then unshadow the guest pages.
Interesting.  I would expect this to be painful on 64-bit windows, which
writes more intermediate values to pagetables, but the numbers disagree.
Maybe it never writes three values back to back.  Certainly a better way
of detecting when pagetables are reused as data pages is welcome, though
we''ve been bitten here before.
> The idea is copied from the KVM side. And since the idea has
> overlapped with the now optimization in the now Xen shadow code, so
> the patch removes the optimization part which when the guest writes 2
> continuously 0 to the same page, then unshadow the guest page.
So to summarize, seven of your performance tests get slightly worse, two
get slightly better and two (x64 linux kernel build times and iperf) get
markedly better.  Do you have confidence intervals for the measurements,
by the way?

Do your HVM linux guests have PV drivers?  If not, does using PV drivers
make a difference to iperf?
> +    if ( v->arch.paging.shadow.last_emulated_gfn => +               
mfn_to_gfn(v->domain, gmfn) )
Is there a reason for tracking this a a GFN instead of an MFN?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>-----Original Message-----
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
>Sent: 2008年1月23日 17:52
>To: Xin, Xiaohui
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel][PATCH]Provide 3 times continously writes check and
>unshadow the guest page
>
>Hi,
>
>At 13:41 +0800 on 23 Jan (1201095692), Xin, Xiaohui wrote:
>> The patch checks if the guest continuously writes 3 times to the same
>> guest page table pages, if yes, then unshadow the guest pages.
>
>Interesting.  I would expect this to be painful on 64-bit windows, which
>writes more intermediate values to pagetables, but the numbers disagree.
>Maybe it never writes three values back to back.  Certainly a better way
>of detecting when pagetables are reused as data pages is welcome, though
>we''ve been bitten here before.
>
>> The idea is copied from the KVM side. And since the idea has
>> overlapped with the now optimization in the now Xen shadow code, so
>> the patch removes the optimization part which when the guest writes 2
>> continuously 0 to the same page, then unshadow the guest page.
>
>So to summarize, seven of your performance tests get slightly worse, two
>get slightly better and two (x64 linux kernel build times and iperf) get
>markedly better.  Do you have confidence intervals for the measurements,
>by the way?
>
Yes, we have confidence with the measurements, since it''s always the
average number of 5 or 6 times.
>Do your HVM linux guests have PV drivers?  If not, does using PV drivers
>make a difference to iperf?
>
No, the guests we used do not have PV drivers. What your concerns are on this
point?
>> +    if ( v->arch.paging.shadow.last_emulated_gfn =>> +       
mfn_to_gfn(v->domain, gmfn) )
>
>Is there a reason for tracking this a a GFN instead of an MFN?
>
You''re right, guest gfn always have one gmfn, so we have over-thought
on it.

Then, how about your final opinion about the patch? We did not see it clearly
from your reply. :-(
>Cheers,
>
>Tim.
>
>--
>Tim Deegan <Tim.Deegan@citrix.com>
>Principal Software Engineer, Citrix Systems (R&D) Ltd.
>[Company #02300071, SL9 0DZ, UK.]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 10:19 +0800 on 24 Jan (1201169995), Xin, Xiaohui
wrote:
> >So to summarize, seven of your performance tests get slightly worse,
two
> >get slightly better and two (x64 linux kernel build times and iperf)
get
> >markedly better.  Do you have confidence intervals for the
measurements,
> >by the way?
> >
> Yes, we have confidence with the measurements, since it''s always
the
> average number of 5 or 6 times.
What I was asking for is some measurement of the variation among those 5
or 6 times.  Ideally a metric of the statistical significance of the
change, but at least some idea of whether I should take a ~1% change
seriously.  For example, if a kernel build changes from 1m44s to 1m43s,
that''s maybe interesting if all 6 were exactly the same, but really not
if there was about 10sec variation among them.
> >Do your HVM linux guests have PV drivers?  If not, does using PV
drivers
> >make a difference to iperf?
> >
> No, the guests we used do not have PV drivers. What your concerns are
> on this point?
Just wondering whether using PV drivers results in better behaviour wrt
shadowed data pages.
> Then, how about your final opinion about the patch? We did not see it
> clearly from your reply. :-(
I''m undecided about it.  The measurements you gave look like it fixes
one particularly bad case very well, but makes overall performance
worse.  In that case, I''m wondering whether there might be a better way
of fixing the network-buffer issue without degrading general
performance.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>-----Original Message-----
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
>Sent: 2008年1月24日 17:49
>To: Xin, Xiaohui
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel][PATCH]Provide 3 times continously writes check and
>unshadow the guest page
>
>At 10:19 +0800 on 24 Jan (1201169995), Xin, Xiaohui wrote:
>> >So to summarize, seven of your performance tests get slightly
worse, two
>> >get slightly better and two (x64 linux kernel build times and
iperf) get
>> >markedly better.  Do you have confidence intervals for the
measurements,
>> >by the way?
>> >
>> Yes, we have confidence with the measurements, since it''s
always the
>> average number of 5 or 6 times.
>
>What I was asking for is some measurement of the variation among those 5
>or 6 times.  Ideally a metric of the statistical significance of the
>change, but at least some idea of whether I should take a ~1% change
>seriously.  For example, if a kernel build changes from 1m44s to 1m43s,
>that''s maybe interesting if all 6 were exactly the same, but really
not
>if there was about 10sec variation among them.
>
I have attached the original test result from our colleague who have done all
the test, the performance data you have seen is concluded from it, just for your
reference, and if you still have some other concerns or worries about the data,
let us dig into it.
>> >Do your HVM linux guests have PV drivers?  If not, does using PV
drivers
>> >make a difference to iperf?
>> >
>> No, the guests we used do not have PV drivers. What your concerns are
>> on this point?
>
>Just wondering whether using PV drivers results in better behaviour wrt
>shadowed data pages.
>
>> Then, how about your final opinion about the patch? We did not see it
>> clearly from your reply. :-(
>
>I''m undecided about it.  The measurements you gave look like it
fixes
>one particularly bad case very well, but makes overall performance
>worse.  In that case, I''m wondering whether there might be a better
way
>of fixing the network-buffer issue without degrading general
>performance.
>
It fixed one bad case, and it improved the kernel build for ia32e Linux guest to
9.5%.
>Cheers,
>
>Tim.
>
>--
>Tim Deegan <Tim.Deegan@citrix.com>
>Principal Software Engineer, Citrix Systems (R&D) Ltd.
>[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

At 09:49 +0000 on 24 Jan (1201168161), Tim Deegan wrote:
> The measurements you gave look like it fixes
> one particularly bad case very well, but makes overall performance
> worse.  In that case, I''m wondering whether there might be a
better way
> of fixing the network-buffer issue without degrading general
> performance.
What''s the trade-off if you use numbers other than three in this
heuristic?  Does a higher number still detetct data pages without
harming the general performance so much?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

We have tried 4 other than 3 before just for Kernel build test, and we just
noticed it will not improve the kernel build for ia32e guest so much. But at
that time we didn''t test for other benchmark. So we will have another
try to see whether high numbers will be helpful. Thanks.

Thanks
Xiaohui 
>-----Original Message-----
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
>Sent: 2008年1月28日 17:49
>To: Xin, Xiaohui
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel][PATCH]Provide 3 times continously writes check and
>unshadow the guest page
>
>Hi,
>
>At 09:49 +0000 on 24 Jan (1201168161), Tim Deegan wrote:
>> The measurements you gave look like it fixes
>> one particularly bad case very well, but makes overall performance
>> worse.  In that case, I''m wondering whether there might be a
better way
>> of fixing the network-buffer issue without degrading general
>> performance.
>
>What''s the trade-off if you use numbers other than three in this
>heuristic?  Does a higher number still detetct data pages without
>harming the general performance so much?
>
>Cheers,
>
>Tim.
>
>--
>Tim Deegan <Tim.Deegan@citrix.com>
>Principal Software Engineer, Citrix Systems (R&D) Ltd.
>[Company #02300071, SL9 0DZ, UK.]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Tim Deegan
>Sent: 2008年1月28日 17:49
>
>Hi,
>
>At 09:49 +0000 on 24 Jan (1201168161), Tim Deegan wrote:
>> The measurements you gave look like it fixes
>> one particularly bad case very well, but makes overall performance
>> worse.  In that case, I''m wondering whether there might be a 
>better way
>> of fixing the network-buffer issue without degrading general
>> performance.
>
>What''s the trade-off if you use numbers other than three in this
>heuristic?  Does a higher number still detetct data pages without
>harming the general performance so much?
>
>Cheers,
>
>Tim.
>
I noted that existing early unshadow heuristic has one exception
on top level page table. I guess the reason is to avoid incorrect
unshadow on top level which implicates whole address space
being unshadowed with heavy overhead to be re-shadowed later.
Also if top level page table is pointed by current guest CR3, it''s
sure not an indicator for unshadow.

But now I''m considering whether we can release that check on
top level page table if it''s not pinned by current guest CR3. Take
Xiaohui''s iperf case for example, she found incorrectly shadowed
pages are mostly L2 pages on 32 and 32pae. So the point is 
whether OS may update top level page tables continuously like 
by 2 continuous zero writes or 3 continuous writes:

    * At fork, parent mappings are modified to RO, but normally
on leaf entry. Child page table is not active yet at mm clone

    * At exit, normally user VMAs are free-ed one by one and top
level page is unlikey to be seen continuous writes. After user
VMAs are free-ed, it''s easy to free whole pgd directly without 
further touch. That''s one corner case Xiaohui''s patch tries to
solve when it''s used later as data page.

    * OS may also put some special pattern in batch style
like from swap thread. But reasonably those patterns are put
in leaf entries, and continuous touch on top level is less likely.

Please correct me since definitely I may miss important OS
behaviors. If above assumption is true, we may use a heuristic
combo on top of Xiaohui''s patch:
    <3 continuous writes on INACTIVE top level page table>

I guess thrash results in Xiaohui''s test results may come from
such heuristics applied to leaf pages also which is too aggre-
sive in same cases. But for top level pages, we may be able
to catch up one heuristics then... :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

At 15:14 +0800 on 31 Jan (1201792446), Tian, Kevin
wrote:
> I noted that existing early unshadow heuristic has one exception
> on top level page table. I guess the reason is to avoid incorrect
> unshadow on top level which implicates whole address space
> being unshadowed with heavy overhead to be re-shadowed later.
> Also if top level page table is pointed by current guest CR3, it''s
> sure not an indicator for unshadow.
Actually that''s a hangover from the original version of this heuristic,
where we special-cased the top-level shadows by zeroing out non-xen
entries, even if they couldn''t be unshadowed.  The current version has
lost the special-case handling but not replaced it with anything.  Well
spotted - I''ll fix it.
> But now I''m considering whether we can release that check on
> top level page table if it''s not pinned by current guest CR3. Take
> Xiaohui''s iperf case for example, she found incorrectly shadowed
> pages are mostly L2 pages on 32 and 32pae.
Interesting!
> I guess thrash results in Xiaohui''s test results may come from
> such heuristics applied to leaf pages also which is too aggre-
> sive in same cases. But for top level pages, we may be able
> to catch up one heuristics then... :-)
The unshadow-after-N-writes heuristic is designed to catch pagetables
getting reused as data pages, and the existing early-unshadow heuristic
is designed to spot process teardown, so there''s possibly value in
having both.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel