Andres Lagar-Cavilla
2012-Jan-07 03:51 UTC
Re: [PATCH 2 of 2] xenpaging:change page-in process to speed up page-in in xenpaging
> Date: Thu, 5 Jan 2012 09:05:16 +0000 > From: Tim Deegan <tim@xen.org> > To: hongkaixing@huawei.com > Cc: Olaf Hering <olaf@aepfle.de>, xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] [PATCH 2 of 2] xenpaging:change page-in > process to speed up page-in in xenpaging > Message-ID: <20120105090516.GE15595@ocelot.phlegethon.org> > Content-Type: text/plain; charset=iso-8859-1 > > Hello, > > At 03:50 +0000 on 05 Jan (1325735430), hongkaixing@huawei.com wrote: >> xenpaging:change page-in process to speed up page-in in xenpaging >> In this patch,we change the page-in process.Firstly,we add a new >> function paging_in_trigger_sync >> to handle page-in requests directly.and when the requests'' count is up >> to 32,then handle them >> batchly;Most importantly,we use an increasing gfn to test_bit,which >> saves much time. >> In p2m.c,we changes p2m_mem_paging_populate() to return a value; >> The following is a xenpaging test on suse11-64 with 4G memory. >> >> Nums of page_out pages Page out time Page in time(in unstable code) Page >> in time(apply this patch) >> 512M(131072) 2.6s 540s 4.7s >> 2G(524288) 15.5s 2088s 17.7s >> > > Thanks for the patch! That''s an impressive difference. You''re changing > quite a few things in this patch, though. Can you send them as separate > patches so they can be reviewed one at a time? Is one of them in > particular making the difference? I suspect it''s mostly the change to > test_bit(), and the rest is not necessary.Second that, on all counts. Impressive numbers, and, a bit puzzled as to what actually made the difference. I would also like to see changes to xenpaging teased out from changes to the hypervisor. I''ve been sitting on a patch to page-in synchronously, which shortcuts even more aggressively the page-in path: instead of calling populate, we go straight into paging_load. This does not necessitate an additional domctl, and would save even more hypervisor<->pager control round-trips. Do you foresee any conflicts with your current approach? Thanks! Andres> > Cheers, > > Tim. > >> Signed-off-by??hongkaixing<hongkaixing@huawei.com>,shizhen<bicky.shi@huawei.com> >> >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xc_mem_paging.c >> --- a/tools/libxc/xc_mem_paging.c Thu Dec 29 17:08:24 2011 +0800 >> +++ b/tools/libxc/xc_mem_paging.c Thu Dec 29 19:41:39 2011 +0800 >> @@ -73,6 +73,13 @@ >> NULL, NULL, gfn); >> } >> >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id, unsigned >> long gfn) >> +{ >> + return xc_mem_event_control(xch, domain_id, >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN, >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING, >> + NULL, NULL, gfn); >> +} >> >> /* >> * Local variables: >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xenctrl.h >> --- a/tools/libxc/xenctrl.h Thu Dec 29 17:08:24 2011 +0800 >> +++ b/tools/libxc/xenctrl.h Thu Dec 29 19:41:39 2011 +0800 >> @@ -1841,6 +1841,7 @@ >> int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, unsigned >> long gfn); >> int xc_mem_paging_resume(xc_interface *xch, domid_t domain_id, >> unsigned long gfn); >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id,unsigned long >> gfn); >> >> int xc_mem_access_enable(xc_interface *xch, domid_t domain_id, >> void *shared_page, void *ring_page); >> diff -r 052727b8165c -r 978daceef147 tools/xenpaging/xenpaging.c >> --- a/tools/xenpaging/xenpaging.c Thu Dec 29 17:08:24 2011 +0800 >> +++ b/tools/xenpaging/xenpaging.c Thu Dec 29 19:41:39 2011 +0800 >> @@ -594,6 +594,13 @@ >> return ret; >> } >> >> +static int paging_in_trigger_sync(xenpaging_t *paging,unsigned long >> gfn) >> +{ >> + int rc = 0; >> + rc = xc_mem_paging_in(paging->xc_handle, >> paging->mem_event.domain_id,gfn); >> + return rc; >> +} > > This function is > >> + >> int main(int argc, char *argv[]) >> { >> struct sigaction act; >> @@ -605,6 +612,9 @@ >> int i; >> int rc = -1; >> int rc1; >> + int request_count = 0; >> + unsigned long page_in_start_gfn = 0; >> + unsigned long real_page = 0; >> xc_interface *xch; >> >> int open_flags = O_CREAT | O_TRUNC | O_RDWR; >> @@ -773,24 +783,51 @@ >> /* Write all pages back into the guest */ >> if ( interrupted == SIGTERM || interrupted == SIGINT ) >> { >> - int num = 0; >> + request_count = 0; >> for ( i = 0; i < paging->domain_info->max_pages; i++ ) >> { >> - if ( test_bit(i, paging->bitmap) ) >> + real_page = i + page_in_start_gfn; >> + real_page %= paging->domain_info->max_pages; >> + if ( test_bit(real_page, paging->bitmap) ) >> { >> - paging->pagein_queue[num] = i; >> - num++; >> - if ( num == XENPAGING_PAGEIN_QUEUE_SIZE ) >> - break; >> + rc = paging_in_trigger_sync(paging,real_page); >> + if ( 0 == rc ) >> + { >> + request_count++; >> + /* If page_in requests up to 32 then handle >> them */ >> + if( request_count >= 32 ) >> + { >> + page_in_start_gfn=real_page + 1; >> + break; >> + } >> + } >> + else >> + { >> + /* If IO ring is full then handle requests to >> free space */ >> + if( ENOSPC == errno ) >> + { >> + page_in_start_gfn = real_page; >> + break; >> + } >> + /* If p2mt is not p2m_is_paging,then clear >> bitmap; >> + * e.g. a page is paged then it is dropped by >> balloon. >> + */ >> + else if ( EINVAL == errno ) >> + { >> + clear_bit(i,paging->bitmap); >> + policy_notify_paged_in(i); >> + } >> + /* If hypercall fails then go to teardown >> xenpaging */ >> + else >> + { >> + ERROR("Error paging in page"); >> + goto out; >> + } >> + } >> } >> } >> - /* >> - * One more round if there are still pages to process. >> - * If no more pages to process, exit loop. >> - */ >> - if ( num ) >> - page_in_trigger(); >> - else if ( i == paging->domain_info->max_pages ) >> + if( (i==paging->domain_info->max_pages) && >> + >> !RING_HAS_UNCONSUMED_REQUESTS(&paging->mem_event.back_ring) ) >> break; >> } >> else >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/mem_paging.c >> --- a/xen/arch/x86/mm/mem_paging.c Thu Dec 29 17:08:24 2011 +0800 >> +++ b/xen/arch/x86/mm/mem_paging.c Thu Dec 29 19:41:39 2011 +0800 >> @@ -57,7 +57,14 @@ >> return 0; >> } >> break; >> - >> + >> + case XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN: >> + { >> + unsigned long gfn = mec->gfn; >> + return p2m_mem_paging_populate(d, gfn); >> + } >> + break; >> + >> default: >> return -ENOSYS; >> break; >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/p2m.c >> --- a/xen/arch/x86/mm/p2m.c Thu Dec 29 17:08:24 2011 +0800 >> +++ b/xen/arch/x86/mm/p2m.c Thu Dec 29 19:41:39 2011 +0800 >> @@ -874,7 +874,7 @@ >> * already sent to the pager. In this case the caller has to try again >> until the >> * gfn is fully paged in again. >> */ >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn) >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn) >> { >> struct vcpu *v = current; >> mem_event_request_t req; >> @@ -882,10 +882,12 @@ >> p2m_access_t a; >> mfn_t mfn; >> struct p2m_domain *p2m = p2m_get_hostp2m(d); >> + int ret; >> >> /* Check that there''s space on the ring for this request */ >> + ret = -ENOSPC; >> if ( mem_event_check_ring(d, &d->mem_paging) ) >> - return; >> + goto out; >> >> memset(&req, 0, sizeof(req)); >> req.type = MEM_EVENT_TYPE_PAGING; >> @@ -905,19 +907,27 @@ >> } >> p2m_unlock(p2m); >> >> + ret = -EINVAL; >> /* Pause domain if request came from guest and gfn has paging type >> */ >> - if ( p2m_is_paging(p2mt) && v->domain->domain_id == d->domain_id ) >> + if ( p2m_is_paging(p2mt) && v->domain->domain_id == d->domain_id ) >> { >> vcpu_pause_nosync(v); >> req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED; >> } >> /* No need to inform pager if the gfn is not in the page-out path >> */ >> - else if ( p2mt != p2m_ram_paging_out && p2mt != p2m_ram_paged ) >> + else if ( p2mt == p2m_ram_paging_in_start || p2mt =>> p2m_ram_paging_in ) >> { >> /* gfn is already on its way back and vcpu is not paused */ >> mem_event_put_req_producers(&d->mem_paging); >> - return; >> + return 0; >> } >> + else if ( !p2m_is_paging(p2mt) ) >> + { >> + /* please clear the bit in paging->bitmap; */ >> + mem_event_put_req_producers(&d->mem_paging); >> + goto out; >> + } >> + >> >> /* Send request to pager */ >> req.gfn = gfn; >> @@ -925,8 +935,13 @@ >> req.vcpu_id = v->vcpu_id; >> >> mem_event_put_request(d, &d->mem_paging, &req); >> + >> + ret = 0; >> + out: >> + return ret; >> } >> >> + >> /** >> * p2m_mem_paging_prep - Allocate a new page for the guest >> * @d: guest domain >> diff -r 052727b8165c -r 978daceef147 xen/include/asm-x86/p2m.h >> --- a/xen/include/asm-x86/p2m.h Thu Dec 29 17:08:24 2011 +0800 >> +++ b/xen/include/asm-x86/p2m.h Thu Dec 29 19:41:39 2011 +0800 >> @@ -485,7 +485,7 @@ >> /* Tell xenpaging to drop a paged out frame */ >> void p2m_mem_paging_drop_page(struct domain *d, unsigned long gfn); >> /* Start populating a paged out frame */ >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn); >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn); >> /* Prepare the p2m for paging a frame in */ >> int p2m_mem_paging_prep(struct domain *d, unsigned long gfn); >> /* Resume normal operation (in case a domain was paused) */ >> diff -r 052727b8165c -r 978daceef147 xen/include/public/domctl.h >> --- a/xen/include/public/domctl.h Thu Dec 29 17:08:24 2011 +0800 >> +++ b/xen/include/public/domctl.h Thu Dec 29 19:41:39 2011 +0800 >> @@ -721,6 +721,7 @@ >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_EVICT 3 >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_PREP 4 >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_RESUME 5 >> +#define XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN 6 >> >> /* >> * Access permissions. >> > >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > > > ------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > > End of Xen-devel Digest, Vol 83, Issue 39 > ***************************************** >
Hongkaixing
2012-Jan-07 08:38 UTC
Re: [PATCH 2 of 2] xenpaging:change page-in process to speed up page-in in xenpaging
> -----Original Message----- > From: Andres Lagar-Cavilla [mailto:andres@lagarcavilla.org] > Sent: Saturday, January 07, 2012 11:52 AM > To: xen-devel@lists.xensource.com > Cc: tim@xen.org; olaf@aepfle.de; hongkaixing@huawei.com > Subject: Re: [PATCH 2 of 2] xenpaging:change page-in process to speed up page-in in xenpaging > > > Date: Thu, 5 Jan 2012 09:05:16 +0000 > > From: Tim Deegan <tim@xen.org> > > To: hongkaixing@huawei.com > > Cc: Olaf Hering <olaf@aepfle.de>, xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] [PATCH 2 of 2] xenpaging:change page-in > > process to speed up page-in in xenpaging > > Message-ID: <20120105090516.GE15595@ocelot.phlegethon.org> > > Content-Type: text/plain; charset=iso-8859-1 > > > > Hello, > > > > At 03:50 +0000 on 05 Jan (1325735430), hongkaixing@huawei.com wrote: > >> xenpaging:change page-in process to speed up page-in in xenpaging > >> In this patch,we change the page-in process.Firstly,we add a new > >> function paging_in_trigger_sync > >> to handle page-in requests directly.and when the requests'' count is up > >> to 32,then handle them > >> batchly;Most importantly,we use an increasing gfn to test_bit,which > >> saves much time. > >> In p2m.c,we changes p2m_mem_paging_populate() to return a value; > >> The following is a xenpaging test on suse11-64 with 4G memory. > >> > >> Nums of page_out pages Page out time Page in time(in unstable code) Page > >> in time(apply this patch) > >> 512M(131072) 2.6s 540s 4.7s > >> 2G(524288) 15.5s 2088s 17.7s > >> > > > > Thanks for the patch! That''s an impressive difference. You''re changing > > quite a few things in this patch, though. Can you send them as separate > > patches so they can be reviewed one at a time? Is one of them in > > particular making the difference? I suspect it''s mostly the change to > > test_bit(), and the rest is not necessary. > > Second that, on all counts. Impressive numbers, and, a bit puzzled as to > what actually made the difference.Take paging 512M as a example, before using this patch, it spends 540s in page-in process, after using this patch, only 4.7 s> > I would also like to see changes to xenpaging teased out from changes to > the hypervisor. > > I''ve been sitting on a patch to page-in synchronously, which shortcuts > even more aggressively the page-in path: instead of calling populate, we > go straight into paging_load. This does not necessitate an additional > domctl, and would save even more hypervisor<->pager control round-trips.Would you mind showing your details? I see, I have try this so. My method is : When test the page is paged out, then populate directly without using event ring. Problem is : There may be some concurrency problems, and some strange blue screens.> Do you foresee any conflicts with your current approach?Our approach is stable till now.> > Thanks! > Andres > > > > > Cheers, > > > > Tim. > > > >> Signed-off-by??hongkaixing<hongkaixing@huawei.com>,shizhen<bicky.shi@huawei.com> > >> > >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xc_mem_paging.c > >> --- a/tools/libxc/xc_mem_paging.c Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/tools/libxc/xc_mem_paging.c Thu Dec 29 19:41:39 2011 +0800 > >> @@ -73,6 +73,13 @@ > >> NULL, NULL, gfn); > >> } > >> > >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id, unsigned > >> long gfn) > >> +{ > >> + return xc_mem_event_control(xch, domain_id, > >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN, > >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING, > >> + NULL, NULL, gfn); > >> +} > >> > >> /* > >> * Local variables: > >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xenctrl.h > >> --- a/tools/libxc/xenctrl.h Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/tools/libxc/xenctrl.h Thu Dec 29 19:41:39 2011 +0800 > >> @@ -1841,6 +1841,7 @@ > >> int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, unsigned > >> long gfn); > >> int xc_mem_paging_resume(xc_interface *xch, domid_t domain_id, > >> unsigned long gfn); > >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id,unsigned long > >> gfn); > >> > >> int xc_mem_access_enable(xc_interface *xch, domid_t domain_id, > >> void *shared_page, void *ring_page); > >> diff -r 052727b8165c -r 978daceef147 tools/xenpaging/xenpaging.c > >> --- a/tools/xenpaging/xenpaging.c Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/tools/xenpaging/xenpaging.c Thu Dec 29 19:41:39 2011 +0800 > >> @@ -594,6 +594,13 @@ > >> return ret; > >> } > >> > >> +static int paging_in_trigger_sync(xenpaging_t *paging,unsigned long > >> gfn) > >> +{ > >> + int rc = 0; > >> + rc = xc_mem_paging_in(paging->xc_handle, > >> paging->mem_event.domain_id,gfn); > >> + return rc; > >> +} > > > > This function is > > > >> + > >> int main(int argc, char *argv[]) > >> { > >> struct sigaction act; > >> @@ -605,6 +612,9 @@ > >> int i; > >> int rc = -1; > >> int rc1; > >> + int request_count = 0; > >> + unsigned long page_in_start_gfn = 0; > >> + unsigned long real_page = 0; > >> xc_interface *xch; > >> > >> int open_flags = O_CREAT | O_TRUNC | O_RDWR; > >> @@ -773,24 +783,51 @@ > >> /* Write all pages back into the guest */ > >> if ( interrupted == SIGTERM || interrupted == SIGINT ) > >> { > >> - int num = 0; > >> + request_count = 0; > >> for ( i = 0; i < paging->domain_info->max_pages; i++ ) > >> { > >> - if ( test_bit(i, paging->bitmap) ) > >> + real_page = i + page_in_start_gfn; > >> + real_page %= paging->domain_info->max_pages; > >> + if ( test_bit(real_page, paging->bitmap) ) > >> { > >> - paging->pagein_queue[num] = i; > >> - num++; > >> - if ( num == XENPAGING_PAGEIN_QUEUE_SIZE ) > >> - break; > >> + rc = paging_in_trigger_sync(paging,real_page); > >> + if ( 0 == rc ) > >> + { > >> + request_count++; > >> + /* If page_in requests up to 32 then handle > >> them */ > >> + if( request_count >= 32 ) > >> + { > >> + page_in_start_gfn=real_page + 1; > >> + break; > >> + } > >> + } > >> + else > >> + { > >> + /* If IO ring is full then handle requests to > >> free space */ > >> + if( ENOSPC == errno ) > >> + { > >> + page_in_start_gfn = real_page; > >> + break; > >> + } > >> + /* If p2mt is not p2m_is_paging,then clear > >> bitmap; > >> + * e.g. a page is paged then it is dropped by > >> balloon. > >> + */ > >> + else if ( EINVAL == errno ) > >> + { > >> + clear_bit(i,paging->bitmap); > >> + policy_notify_paged_in(i); > >> + } > >> + /* If hypercall fails then go to teardown > >> xenpaging */ > >> + else > >> + { > >> + ERROR("Error paging in page"); > >> + goto out; > >> + } > >> + } > >> } > >> } > >> - /* > >> - * One more round if there are still pages to process. > >> - * If no more pages to process, exit loop. > >> - */ > >> - if ( num ) > >> - page_in_trigger(); > >> - else if ( i == paging->domain_info->max_pages ) > >> + if( (i==paging->domain_info->max_pages) && > >> + > >> !RING_HAS_UNCONSUMED_REQUESTS(&paging->mem_event.back_ring) ) > >> break; > >> } > >> else > >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/mem_paging.c > >> --- a/xen/arch/x86/mm/mem_paging.c Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/xen/arch/x86/mm/mem_paging.c Thu Dec 29 19:41:39 2011 +0800 > >> @@ -57,7 +57,14 @@ > >> return 0; > >> } > >> break; > >> - > >> + > >> + case XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN: > >> + { > >> + unsigned long gfn = mec->gfn; > >> + return p2m_mem_paging_populate(d, gfn); > >> + } > >> + break; > >> + > >> default: > >> return -ENOSYS; > >> break; > >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/p2m.c > >> --- a/xen/arch/x86/mm/p2m.c Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/xen/arch/x86/mm/p2m.c Thu Dec 29 19:41:39 2011 +0800 > >> @@ -874,7 +874,7 @@ > >> * already sent to the pager. In this case the caller has to try again > >> until the > >> * gfn is fully paged in again. > >> */ > >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn) > >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn) > >> { > >> struct vcpu *v = current; > >> mem_event_request_t req; > >> @@ -882,10 +882,12 @@ > >> p2m_access_t a; > >> mfn_t mfn; > >> struct p2m_domain *p2m = p2m_get_hostp2m(d); > >> + int ret; > >> > >> /* Check that there''s space on the ring for this request */ > >> + ret = -ENOSPC; > >> if ( mem_event_check_ring(d, &d->mem_paging) ) > >> - return; > >> + goto out; > >> > >> memset(&req, 0, sizeof(req)); > >> req.type = MEM_EVENT_TYPE_PAGING; > >> @@ -905,19 +907,27 @@ > >> } > >> p2m_unlock(p2m); > >> > >> + ret = -EINVAL; > >> /* Pause domain if request came from guest and gfn has paging type > >> */ > >> - if ( p2m_is_paging(p2mt) && v->domain->domain_id == d->domain_id ) > >> + if ( p2m_is_paging(p2mt) && v->domain->domain_id == d->domain_id ) > >> { > >> vcpu_pause_nosync(v); > >> req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED; > >> } > >> /* No need to inform pager if the gfn is not in the page-out path > >> */ > >> - else if ( p2mt != p2m_ram_paging_out && p2mt != p2m_ram_paged ) > >> + else if ( p2mt == p2m_ram_paging_in_start || p2mt => >> p2m_ram_paging_in ) > >> { > >> /* gfn is already on its way back and vcpu is not paused */ > >> mem_event_put_req_producers(&d->mem_paging); > >> - return; > >> + return 0; > >> } > >> + else if ( !p2m_is_paging(p2mt) ) > >> + { > >> + /* please clear the bit in paging->bitmap; */ > >> + mem_event_put_req_producers(&d->mem_paging); > >> + goto out; > >> + } > >> + > >> > >> /* Send request to pager */ > >> req.gfn = gfn; > >> @@ -925,8 +935,13 @@ > >> req.vcpu_id = v->vcpu_id; > >> > >> mem_event_put_request(d, &d->mem_paging, &req); > >> + > >> + ret = 0; > >> + out: > >> + return ret; > >> } > >> > >> + > >> /** > >> * p2m_mem_paging_prep - Allocate a new page for the guest > >> * @d: guest domain > >> diff -r 052727b8165c -r 978daceef147 xen/include/asm-x86/p2m.h > >> --- a/xen/include/asm-x86/p2m.h Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/xen/include/asm-x86/p2m.h Thu Dec 29 19:41:39 2011 +0800 > >> @@ -485,7 +485,7 @@ > >> /* Tell xenpaging to drop a paged out frame */ > >> void p2m_mem_paging_drop_page(struct domain *d, unsigned long gfn); > >> /* Start populating a paged out frame */ > >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn); > >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn); > >> /* Prepare the p2m for paging a frame in */ > >> int p2m_mem_paging_prep(struct domain *d, unsigned long gfn); > >> /* Resume normal operation (in case a domain was paused) */ > >> diff -r 052727b8165c -r 978daceef147 xen/include/public/domctl.h > >> --- a/xen/include/public/domctl.h Thu Dec 29 17:08:24 2011 +0800 > >> +++ b/xen/include/public/domctl.h Thu Dec 29 19:41:39 2011 +0800 > >> @@ -721,6 +721,7 @@ > >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_EVICT 3 > >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_PREP 4 > >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_RESUME 5 > >> +#define XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN 6 > >> > >> /* > >> * Access permissions. > >> > > > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > > > > > > > ------------------------------ > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > > End of Xen-devel Digest, Vol 83, Issue 39 > > ***************************************** > >
Andres Lagar-Cavilla
2012-Jan-10 03:33 UTC
Re: [PATCH 2 of 2] xenpaging:change page-in process to speed up page-in in xenpaging
> >> -----Original Message----- >> From: Andres Lagar-Cavilla [mailto:andres@lagarcavilla.org] >> Sent: Saturday, January 07, 2012 11:52 AM >> To: xen-devel@lists.xensource.com >> Cc: tim@xen.org; olaf@aepfle.de; hongkaixing@huawei.com >> Subject: Re: [PATCH 2 of 2] xenpaging:change page-in process to speed up >> page-in in xenpaging >> >> > Date: Thu, 5 Jan 2012 09:05:16 +0000 >> > From: Tim Deegan <tim@xen.org> >> > To: hongkaixing@huawei.com >> > Cc: Olaf Hering <olaf@aepfle.de>, xen-devel@lists.xensource.com >> > Subject: Re: [Xen-devel] [PATCH 2 of 2] xenpaging:change page-in >> > process to speed up page-in in xenpaging >> > Message-ID: <20120105090516.GE15595@ocelot.phlegethon.org> >> > Content-Type: text/plain; charset=iso-8859-1 >> > >> > Hello, >> > >> > At 03:50 +0000 on 05 Jan (1325735430), hongkaixing@huawei.com wrote: >> >> xenpaging:change page-in process to speed up page-in in xenpaging >> >> In this patch,we change the page-in process.Firstly,we add a new >> >> function paging_in_trigger_sync >> >> to handle page-in requests directly.and when the requests'' count is >> up >> >> to 32,then handle them >> >> batchly;Most importantly,we use an increasing gfn to test_bit,which >> >> saves much time. >> >> In p2m.c,we changes p2m_mem_paging_populate() to return a value; >> >> The following is a xenpaging test on suse11-64 with 4G memory. >> >> >> >> Nums of page_out pages Page out time Page in time(in unstable code) >> Page >> >> in time(apply this patch) >> >> 512M(131072) 2.6s 540s 4.7s >> >> 2G(524288) 15.5s 2088s 17.7s >> >> >> > >> > Thanks for the patch! That''s an impressive difference. You''re >> changing >> > quite a few things in this patch, though. Can you send them as >> separate >> > patches so they can be reviewed one at a time? Is one of them in >> > particular making the difference? I suspect it''s mostly the change to >> > test_bit(), and the rest is not necessary. >> >> Second that, on all counts. Impressive numbers, and, a bit puzzled as to >> what actually made the difference. > > Take paging 512M as a example, before using this patch, it spends 540s in > page-in process, after using this patch, only 4.7 s > >> >> I would also like to see changes to xenpaging teased out from changes to >> the hypervisor. >> >> I''ve been sitting on a patch to page-in synchronously, which shortcuts >> even more aggressively the page-in path: instead of calling populate, we >> go straight into paging_load. This does not necessitate an additional >> domctl, and would save even more hypervisor<->pager control round-trips. > > Would you mind showing your details?I just sent the patch, look for "[PATCH 1 of 2] x86/mm: Allow a page in p2m_ram_paged_out state to be loaded" This should allow xenpaging to populate pages directly in its page-in thread, without needing a populate domctl (or a mmap). Andres> I see, I have try this so. My method is : > When test the page is paged out, then populate directly without using > event ring. > Problem is : There may be some concurrency problems, and some strange blue > screens. > > >> Do you foresee any conflicts with your current approach? > > Our approach is stable till now. >> >> Thanks! >> Andres >> >> > >> > Cheers, >> > >> > Tim. >> > >> >> Signed-off-by??hongkaixing<hongkaixing@huawei.com>,shizhen<bicky.shi@huawei.com> >> >> >> >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xc_mem_paging.c >> >> --- a/tools/libxc/xc_mem_paging.c Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/tools/libxc/xc_mem_paging.c Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -73,6 +73,13 @@ >> >> NULL, NULL, gfn); >> >> } >> >> >> >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id, unsigned >> >> long gfn) >> >> +{ >> >> + return xc_mem_event_control(xch, domain_id, >> >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN, >> >> + XEN_DOMCTL_MEM_EVENT_OP_PAGING, >> >> + NULL, NULL, gfn); >> >> +} >> >> >> >> /* >> >> * Local variables: >> >> diff -r 052727b8165c -r 978daceef147 tools/libxc/xenctrl.h >> >> --- a/tools/libxc/xenctrl.h Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/tools/libxc/xenctrl.h Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -1841,6 +1841,7 @@ >> >> int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, >> unsigned >> >> long gfn); >> >> int xc_mem_paging_resume(xc_interface *xch, domid_t domain_id, >> >> unsigned long gfn); >> >> +int xc_mem_paging_in(xc_interface *xch, domid_t domain_id,unsigned >> long >> >> gfn); >> >> >> >> int xc_mem_access_enable(xc_interface *xch, domid_t domain_id, >> >> void *shared_page, void *ring_page); >> >> diff -r 052727b8165c -r 978daceef147 tools/xenpaging/xenpaging.c >> >> --- a/tools/xenpaging/xenpaging.c Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/tools/xenpaging/xenpaging.c Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -594,6 +594,13 @@ >> >> return ret; >> >> } >> >> >> >> +static int paging_in_trigger_sync(xenpaging_t *paging,unsigned long >> >> gfn) >> >> +{ >> >> + int rc = 0; >> >> + rc = xc_mem_paging_in(paging->xc_handle, >> >> paging->mem_event.domain_id,gfn); >> >> + return rc; >> >> +} >> > >> > This function is >> > >> >> + >> >> int main(int argc, char *argv[]) >> >> { >> >> struct sigaction act; >> >> @@ -605,6 +612,9 @@ >> >> int i; >> >> int rc = -1; >> >> int rc1; >> >> + int request_count = 0; >> >> + unsigned long page_in_start_gfn = 0; >> >> + unsigned long real_page = 0; >> >> xc_interface *xch; >> >> >> >> int open_flags = O_CREAT | O_TRUNC | O_RDWR; >> >> @@ -773,24 +783,51 @@ >> >> /* Write all pages back into the guest */ >> >> if ( interrupted == SIGTERM || interrupted == SIGINT ) >> >> { >> >> - int num = 0; >> >> + request_count = 0; >> >> for ( i = 0; i < paging->domain_info->max_pages; i++ ) >> >> { >> >> - if ( test_bit(i, paging->bitmap) ) >> >> + real_page = i + page_in_start_gfn; >> >> + real_page %= paging->domain_info->max_pages; >> >> + if ( test_bit(real_page, paging->bitmap) ) >> >> { >> >> - paging->pagein_queue[num] = i; >> >> - num++; >> >> - if ( num == XENPAGING_PAGEIN_QUEUE_SIZE ) >> >> - break; >> >> + rc = paging_in_trigger_sync(paging,real_page); >> >> + if ( 0 == rc ) >> >> + { >> >> + request_count++; >> >> + /* If page_in requests up to 32 then handle >> >> them */ >> >> + if( request_count >= 32 ) >> >> + { >> >> + page_in_start_gfn=real_page + 1; >> >> + break; >> >> + } >> >> + } >> >> + else >> >> + { >> >> + /* If IO ring is full then handle requests >> to >> >> free space */ >> >> + if( ENOSPC == errno ) >> >> + { >> >> + page_in_start_gfn = real_page; >> >> + break; >> >> + } >> >> + /* If p2mt is not p2m_is_paging,then clear >> >> bitmap; >> >> + * e.g. a page is paged then it is dropped by >> >> balloon. >> >> + */ >> >> + else if ( EINVAL == errno ) >> >> + { >> >> + clear_bit(i,paging->bitmap); >> >> + policy_notify_paged_in(i); >> >> + } >> >> + /* If hypercall fails then go to teardown >> >> xenpaging */ >> >> + else >> >> + { >> >> + ERROR("Error paging in page"); >> >> + goto out; >> >> + } >> >> + } >> >> } >> >> } >> >> - /* >> >> - * One more round if there are still pages to process. >> >> - * If no more pages to process, exit loop. >> >> - */ >> >> - if ( num ) >> >> - page_in_trigger(); >> >> - else if ( i == paging->domain_info->max_pages ) >> >> + if( (i==paging->domain_info->max_pages) && >> >> + >> >> !RING_HAS_UNCONSUMED_REQUESTS(&paging->mem_event.back_ring) ) >> >> break; >> >> } >> >> else >> >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/mem_paging.c >> >> --- a/xen/arch/x86/mm/mem_paging.c Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/xen/arch/x86/mm/mem_paging.c Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -57,7 +57,14 @@ >> >> return 0; >> >> } >> >> break; >> >> - >> >> + >> >> + case XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN: >> >> + { >> >> + unsigned long gfn = mec->gfn; >> >> + return p2m_mem_paging_populate(d, gfn); >> >> + } >> >> + break; >> >> + >> >> default: >> >> return -ENOSYS; >> >> break; >> >> diff -r 052727b8165c -r 978daceef147 xen/arch/x86/mm/p2m.c >> >> --- a/xen/arch/x86/mm/p2m.c Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/xen/arch/x86/mm/p2m.c Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -874,7 +874,7 @@ >> >> * already sent to the pager. In this case the caller has to try >> again >> >> until the >> >> * gfn is fully paged in again. >> >> */ >> >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn) >> >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn) >> >> { >> >> struct vcpu *v = current; >> >> mem_event_request_t req; >> >> @@ -882,10 +882,12 @@ >> >> p2m_access_t a; >> >> mfn_t mfn; >> >> struct p2m_domain *p2m = p2m_get_hostp2m(d); >> >> + int ret; >> >> >> >> /* Check that there''s space on the ring for this request */ >> >> + ret = -ENOSPC; >> >> if ( mem_event_check_ring(d, &d->mem_paging) ) >> >> - return; >> >> + goto out; >> >> >> >> memset(&req, 0, sizeof(req)); >> >> req.type = MEM_EVENT_TYPE_PAGING; >> >> @@ -905,19 +907,27 @@ >> >> } >> >> p2m_unlock(p2m); >> >> >> >> + ret = -EINVAL; >> >> /* Pause domain if request came from guest and gfn has paging >> type >> >> */ >> >> - if ( p2m_is_paging(p2mt) && v->domain->domain_id =>> d->domain_id ) >> >> + if ( p2m_is_paging(p2mt) && v->domain->domain_id == d->domain_id >> ) >> >> { >> >> vcpu_pause_nosync(v); >> >> req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED; >> >> } >> >> /* No need to inform pager if the gfn is not in the page-out >> path >> >> */ >> >> - else if ( p2mt != p2m_ram_paging_out && p2mt != p2m_ram_paged ) >> >> + else if ( p2mt == p2m_ram_paging_in_start || p2mt =>> >> p2m_ram_paging_in ) >> >> { >> >> /* gfn is already on its way back and vcpu is not paused */ >> >> mem_event_put_req_producers(&d->mem_paging); >> >> - return; >> >> + return 0; >> >> } >> >> + else if ( !p2m_is_paging(p2mt) ) >> >> + { >> >> + /* please clear the bit in paging->bitmap; */ >> >> + mem_event_put_req_producers(&d->mem_paging); >> >> + goto out; >> >> + } >> >> + >> >> >> >> /* Send request to pager */ >> >> req.gfn = gfn; >> >> @@ -925,8 +935,13 @@ >> >> req.vcpu_id = v->vcpu_id; >> >> >> >> mem_event_put_request(d, &d->mem_paging, &req); >> >> + >> >> + ret = 0; >> >> + out: >> >> + return ret; >> >> } >> >> >> >> + >> >> /** >> >> * p2m_mem_paging_prep - Allocate a new page for the guest >> >> * @d: guest domain >> >> diff -r 052727b8165c -r 978daceef147 xen/include/asm-x86/p2m.h >> >> --- a/xen/include/asm-x86/p2m.h Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/xen/include/asm-x86/p2m.h Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -485,7 +485,7 @@ >> >> /* Tell xenpaging to drop a paged out frame */ >> >> void p2m_mem_paging_drop_page(struct domain *d, unsigned long gfn); >> >> /* Start populating a paged out frame */ >> >> -void p2m_mem_paging_populate(struct domain *d, unsigned long gfn); >> >> +int p2m_mem_paging_populate(struct domain *d, unsigned long gfn); >> >> /* Prepare the p2m for paging a frame in */ >> >> int p2m_mem_paging_prep(struct domain *d, unsigned long gfn); >> >> /* Resume normal operation (in case a domain was paused) */ >> >> diff -r 052727b8165c -r 978daceef147 xen/include/public/domctl.h >> >> --- a/xen/include/public/domctl.h Thu Dec 29 17:08:24 2011 +0800 >> >> +++ b/xen/include/public/domctl.h Thu Dec 29 19:41:39 2011 +0800 >> >> @@ -721,6 +721,7 @@ >> >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_EVICT 3 >> >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_PREP 4 >> >> #define XEN_DOMCTL_MEM_EVENT_OP_PAGING_RESUME 5 >> >> +#define XEN_DOMCTL_MEM_EVENT_OP_PAGING_IN 6 >> >> >> >> /* >> >> * Access permissions. >> >> >> > >> >> _______________________________________________ >> >> Xen-devel mailing list >> >> Xen-devel@lists.xensource.com >> >> http://lists.xensource.com/xen-devel >> > >> > >> > >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > Xen-devel mailing list >> > Xen-devel@lists.xensource.com >> > http://lists.xensource.com/xen-devel >> > >> > >> > End of Xen-devel Digest, Vol 83, Issue 39 >> > ***************************************** >> > > > >