Hi, I noticed that parallel scrubbing pages is not efficient due to spinlock contention. (also memory bandwidth?) Actually heap_lock becomes a bottleneck. In my investigation, it often takes 1 millisec just to acquire the lock on ia64 with 8cpus. It''s very wasteful. For example, creating a domain is too slow while scrubbing pages. # xm create vm memory=4000 # xm destroy vm; time xm create vm memory=4000 real 0m6.083s user 0m0.024s sys 0m0.008s After the attached patch is applied, # xm destroy vm; time xm create vm memory=4000 real 0m1.463s user 0m0.132s sys 0m0.028s This workaround is an easy solution but ugly. There must be another good way. Thanks, Kouya Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/8/08 11:01, "Kouya Shimura" <kouya@jp.fujitsu.com> wrote:> I noticed that parallel scrubbing pages is not efficient > due to spinlock contention. (also memory bandwidth?) > Actually heap_lock becomes a bottleneck.I''ve considered the possibility of moving this kind of thing into a service domain. Then it would be visible to the scheduler and be less of a hack. Anyhow, I hadn''t considered the contention for the heap_lock. I think your patch is good and I will apply it for 3.3. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/8/08 11:12, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> I''ve considered the possibility of moving this kind of thing into a service > domain. Then it would be visible to the scheduler and be less of a hack. > > Anyhow, I hadn''t considered the contention for the heap_lock. I think your > patch is good and I will apply it for 3.3.I''ve simplified the patch and then checked it in. There was no need for a new spinlock as it strictly dominated the old one. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kouya Shimura
2008-Aug-11 11:07 UTC
Re: [Xen-devel] [PATCH][RFC] Serialize scrubbing pages
Hi Keir, I''m afraid that cset18305:f10d436d9a9e makes free_domheap_page freeze. Inner loop of page_scrub_softirq which keeps page_scrub_lock takes at least 1 millisec. Is it OK? That''s why I prepare a new spinlock. Thanks, Kouya Keir Fraser writes:> On 11/8/08 11:12, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > > > I''ve considered the possibility of moving this kind of thing into a service > > domain. Then it would be visible to the scheduler and be less of a hack. > > > > Anyhow, I hadn''t considered the contention for the heap_lock. I think your > > patch is good and I will apply it for 3.3. > > I''ve simplified the patch and then checked it in. There was no need for a > new spinlock as it strictly dominated the old one._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/8/08 12:07, "Kouya Shimura" <kouya@jp.fujitsu.com> wrote:> I''m afraid that cset18305:f10d436d9a9e makes free_domheap_page freeze. > Inner loop of page_scrub_softirq which keeps page_scrub_lock takes > at least 1 millisec. Is it OK? > That''s why I prepare a new spinlock.Oh, hmmm, that''s a fair point. I''ll put you extra lock back in, but private to the softirq scrub function. Thanks, Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel