Gianluca Guida
2008-Sep-12 16:51 UTC
[Xen-devel] [PATCH] Allow removing writable mappings from splintered page tables.
Hello, The moving of the pagetable mapping in the linux kernel exposed the fact that under the linux kernel sh_rm_write_access_from_sl1p was always failing. Linux seems to use big pages to access page tables, so we should instruct the shadow code to be able to remove writable mappings from splintered pagetables as well, avoiding using OS heuristic (which were failing in 2.6.27 before George patch, leading to brute-force search at each resync). Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain. Thanks, Gianluca _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Todd Deshane
2008-Sep-12 17:08 UTC
[Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
On Fri, Sep 12, 2008 at 12:51 PM, Gianluca Guida <gianluca.guida@eu.citrix.com> wrote:> Hello, > > The moving of the pagetable mapping in the linux kernel exposed the fact > that under the linux kernel sh_rm_write_access_from_sl1p was always failing. > > Linux seems to use big pages to access page tables, so we should instruct > the shadow code to be able to remove writable mappings from splintered > pagetables as well, avoiding using OS heuristic (which were failing in > 2.6.27 before George patch, leading to brute-force search at each resync). > > Todd: Can you try try the linux-2.6.27 kernbench with the following patch? > Also, are you using ioemu stub-domains? That should also give a performance > gain. >Sure, I can give it a try. Should I also apply George''s patch right? I am not using ioemu stub-domains yet, would you recommend them in a general sense or are they more of a special case? Cheers, Todd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Gianluca Guida
2008-Sep-12 17:14 UTC
[Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
Todd Deshane wrote:>> Todd: Can you try try the linux-2.6.27 kernbench with the following patch? >> Also, are you using ioemu stub-domains? That should also give a performance >> gain. >> > > Sure, I can give it a try. Should I also apply George''s patch right?Yes.> I am not using ioemu stub-domains yet, would you recommend them in a general > sense or are they more of a special case?I think they''re in general recommendable. Both Samuel and Stefano will be able to give more information about it. Thanks, Gianluca _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Samuel Thibault
2008-Sep-12 18:28 UTC
Re: [Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
Gianluca Guida, le Fri 12 Sep 2008 18:14:17 +0100, a écrit :> >I am not using ioemu stub-domains yet, would you recommend them in a > >general > >sense or are they more of a special case? > > I think they''re in general recommendable. Both Samuel and Stefano will > be able to give more information about it.That brings more security and performance, so yes it''s recommended :) Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Todd Deshane
2008-Sep-12 19:52 UTC
[Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
On Fri, Sep 12, 2008 at 1:14 PM, Gianluca Guida <gianluca.guida@eu.citrix.com> wrote:> > > Todd Deshane wrote: >>> >>> Todd: Can you try try the linux-2.6.27 kernbench with the following >>> patch? >>> Also, are you using ioemu stub-domains? That should also give a >>> performance >>> gain. >>> >> >> Sure, I can give it a try. Should I also apply George''s patch right? >Much faster, I''ll also test on debian 4.0 for a regression test. Average Optimal load -j 4 Run (std deviation): Elapsed Time 519.056 (0.098387) User Time 426.226 (0.815555) System Time 92.716 (0.858563) Percent CPU 99 (0) Context Switches 99125.2 (71.3667) Sleeps 31199.2 (41.3062) I also want to look into the ioemu stub domains after reading more about them. I''ll report the debian (older kernel results) when they come out. Todd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Todd Deshane
2008-Sep-12 22:13 UTC
[Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
> I''ll report the debian (older kernel results) when they come out.Roughly the same as before, so the patches look good. HVM guest (Debian 4.0 vanilla 2.6.18.6-amd64) Average Optimal load -j 4 Run (std deviation): Elapsed Time 492.826 (1.89446) User Time 390.316 (0.571341) System Time 101.396 (2.09772) Percent CPU 99 (0) Context Switches 29514.4 (53.8266) Sleeps 37991.4 (572.437) -- Todd Deshane http://todddeshane.net check out our book: http://runningxen.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2008-Sep-15 10:38 UTC
Re: [Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down! If you have time, could you take another 30-second trace with the new changes in, just for fun? I''ll take a quick look and see if there''s any other low-hanging fruit to grab. -George On Fri, Sep 12, 2008 at 8:52 PM, Todd Deshane <deshantm@gmail.com> wrote:> On Fri, Sep 12, 2008 at 1:14 PM, Gianluca Guida > <gianluca.guida@eu.citrix.com> wrote: >> >> >> Todd Deshane wrote: >>>> >>>> Todd: Can you try try the linux-2.6.27 kernbench with the following >>>> patch? >>>> Also, are you using ioemu stub-domains? That should also give a >>>> performance >>>> gain. >>>> >>> >>> Sure, I can give it a try. Should I also apply George''s patch right? >> > > Much faster, I''ll also test on debian 4.0 for a regression test. > > Average Optimal load -j 4 Run (std deviation): > Elapsed Time 519.056 (0.098387) > User Time 426.226 (0.815555) > System Time 92.716 (0.858563) > Percent CPU 99 (0) > Context Switches 99125.2 (71.3667) > Sleeps 31199.2 (41.3062) > > I also want to look into the ioemu stub domains after reading more about them. > > I''ll report the debian (older kernel results) when they come out. > > Todd > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Todd Deshane
2008-Sep-15 16:30 UTC
Re: [Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> And your original numbers showed elapsed time to be 527s for KVM, so > now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the > help tracking this down! >KVM is also working on improved page table algorithms http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html I think the competition is a good thing.> If you have time, could you take another 30-second trace with the new > changes in, just for fun? I''ll take a quick look and see if there''s > any other low-hanging fruit to grab. >Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger. Todd -- Todd Deshane http://todddeshane.net check out our book: http://runningxen.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2008-Sep-15 17:03 UTC
Re: [Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
Heh... the blatant copying is flattering and annoying at the same time. :-) Ah, the beauty of open-source... I''ve got your trace, and I''ll take a look at it tomorrow. Thanks! -George On Mon, Sep 15, 2008 at 5:30 PM, Todd Deshane <deshantm@gmail.com> wrote:> On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: >> And your original numbers showed elapsed time to be 527s for KVM, so >> now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the >> help tracking this down! >> > > KVM is also working on improved page table algorithms > http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html > > I think the competition is a good thing. > >> If you have time, could you take another 30-second trace with the new >> changes in, just for fun? I''ll take a quick look and see if there''s >> any other low-hanging fruit to grab. >> > > Sent the trace to you with another service called sendspace, since, for > some reason, the trace file was much bigger. > > Todd > > -- > Todd Deshane > http://todddeshane.net > check out our book: http://runningxen.com > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2008-Sep-16 13:46 UTC
Re: [Xen-devel] Re: [PATCH] Allow removing writable mappings from splintered page tables.
Hmm, no really obvious low-hanging fruit. The Xen-HVM was about 9% slower than your reported numbers for Xen-PV, and the trace shows that the guest spent about that much inside the hypervisor. The breakdown: * 3.6% propagating page faults to guest * 3.0% pulling through entries from out-of-sync guest pt''s to shadow pagetables * 1.4% Marking pages out of sync (of which 75% was in unsyncs that had to re-sync another page) * 0.9% cr3 switches * 0.9% handling I/O (Rounding may cause the numbers not to add up exactly.) So one of the biggest things, really, is that Linux seems to insist on mapping pages one-at-a-time as they''re demand-faulted, rather than doing a batch of them. Unfortunately, having pages out-of-sync means that we must use the slow propagate path rather than the fast-propagate path, which is at least 25% slower. The only avenues for optimization I can see are: * See if there''s a way to reduce the number of unsyncs that cause resyncs. Allowing more pages to go out-of-sync *might* do this; or it might just shift the same overhead into cr3 switch. * Reduce the time of "hot paths" through the hypervisor by profiling, &c. -George On Mon, Sep 15, 2008 at 6:03 PM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> Heh... the blatant copying is flattering and annoying at the same > time. :-) Ah, the beauty of open-source... > > I''ve got your trace, and I''ll take a look at it tomorrow. Thanks! > > -George > > On Mon, Sep 15, 2008 at 5:30 PM, Todd Deshane <deshantm@gmail.com> wrote: >> On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap >> <George.Dunlap@eu.citrix.com> wrote: >>> And your original numbers showed elapsed time to be 527s for KVM, so >>> now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the >>> help tracking this down! >>> >> >> KVM is also working on improved page table algorithms >> http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html >> >> I think the competition is a good thing. >> >>> If you have time, could you take another 30-second trace with the new >>> changes in, just for fun? I''ll take a quick look and see if there''s >>> any other low-hanging fruit to grab. >>> >> >> Sent the trace to you with another service called sendspace, since, for >> some reason, the trace file was much bigger. >> >> Todd >> >> -- >> Todd Deshane >> http://todddeshane.net >> check out our book: http://runningxen.com >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel