The memory permutation cause a slow down in case of a save/restore (bug 1143). It works better when the mixing is done with 2MB chunks. Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Jean, The patch does fix the bug. Great! I made a test to change xc_hvm_build() to invoke xc_domain_memory_populate_physmap() in the same pfn order of that in the old permute(); then I created an HVM guest, and I met with almost the same slowness in it! Looks the old poor version of permute() can incur high rate of cache miss, hence the slowness after S/R is caused? However, I still have questions: For the bug, I remember the slowness 1) Only happens to HVM guest (PV-guest has not this issue); -- any difference between HVM and PV here?? 2) Only happens to S/R and local non-live migration, but doesn''t happen to local live migration. -- any difference between live and non-live here?? And when we suffer from the slowness, "local live migrating" the HVM guest can make the performance back to normal! Can you reproduce these in your side? If so, can you help to explain them? Many thanks! -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jean Guyader Sent: 2008年3月19日 2:03 To: xen-devel@lists.xensource.com Subject: [Xen-devel] [PATCH] permute with 2MB chunk The memory permutation cause a slow down in case of a save/restore (bug 1143). It works better when the mixing is done with 2MB chunks. Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Also a bit curious whether original intent of permute() still keeps
true on a 2M granularity from this good patch:
/* Need a simple permutation function so that we scan pages in a
pseudo random order, enabling us to get a better estimate of
the domain''s page dirtying rate as we go (there are often
contiguous ranges of pfns that have similar behaviour, and we
want to mix them up. */
And if not, maybe permute() can be removed instead, or with some
counterpart in restore side? :-)
Thanks,
Kevin>From: Cui, Dexuan
>Sent: 2008年3月19日 17:42
>
>Hi Jean,
>The patch does fix the bug. Great!
>
>I made a test to change xc_hvm_build() to invoke
>xc_domain_memory_populate_physmap() in the same pfn order of
>that in the old permute(); then I created an HVM guest, and I
>met with almost the same slowness in it!
>Looks the old poor version of permute() can incur high rate of
>cache miss, hence the slowness after S/R is caused?
>
>However, I still have questions:
>For the bug, I remember the slowness
>1) Only happens to HVM guest (PV-guest has not this issue);
>-- any difference between HVM and PV here??
>2) Only happens to S/R and local non-live migration, but
>doesn''t happen to local live migration. -- any difference
>between live and non-live here??
>And when we suffer from the slowness, "local live migrating"
>the HVM guest can make the performance back to normal!
>
>Can you reproduce these in your side? If so, can you help to
>explain them?
>
>Many thanks!
>
>-- Dexuan
>
>
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>Jean Guyader
>Sent: 2008年3月19日 2:03
>To: xen-devel@lists.xensource.com
>Subject: [Xen-devel] [PATCH] permute with 2MB chunk
>
>
>The memory permutation cause a slow down in case of a
>save/restore (bug
>1143). It works better when the mixing is done with 2MB chunks.
>
>Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 19/3/08 09:42, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> However, I still have questions: > For the bug, I remember the slowness > 1) Only happens to HVM guest (PV-guest has not this issue); -- any > difference between HVM and PV here?? > 2) Only happens to S/R and local non-live migration, but doesn''t happen to > local live migration. -- any difference between live and non-live here?? > And when we suffer from the slowness, "local live migrating" the HVM guest can > make the performance back to normal! > > Can you reproduce these in your side? If so, can you help to explain them?We also tested building an HVM guest with the permuted ordering of pages, versus reverse ordering, versus normal ordering. Only the permuted ordering showed the problem. We assume that the permute() function has an unfortunate interaction with the memory allocator in certain HVM guest OSes, causing poor cache utilisation. The fact that live migration made the bug go away can perhaps be explained by the fact that multiple rounds of page transmission add an extra layer of randomisation to page allocations at the receiver? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> We also tested building an HVM guest with the permuted ordering of > pages, versus reverse ordering, versus normal ordering. Only thepermuted> ordering showed the problem. We assume that the permute() function hasan> unfortunate interaction with the memory allocator in certain HVM guestOSes,> causing poor cache utilisation.It''s still very odd that the permutation fn only seems to effect Linux running as a HVM guest and not as a PV guest. I still think there''s something we''re not quite understanding. Jean: have you definitely verified that building a domain with the permute function does not affect Linux PV guests? Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20/3/08 09:05, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:>> We also tested building an HVM guest with the permuted ordering of >> pages, versus reverse ordering, versus normal ordering. Only the > permuted >> ordering showed the problem. We assume that the permute() function has > an >> unfortunate interaction with the memory allocator in certain HVM guest > OSes, >> causing poor cache utilisation. > > It''s still very odd that the permutation fn only seems to effect Linux > running as a HVM guest and not as a PV guest. I still think there''s > something we''re not quite understanding. > > Jean: have you definitely verified that building a domain with the > permute function does not affect Linux PV guests?Dexuan has also claimed in private email that the 2MB permute function speeds up kernel builds in save-restored HVM guests from 62s to 58s. We don''t know the confidence intervals for those figures though. Still, there''s presumably something rather fragile underlying all this... -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:>> We also tested building an HVM guest with the permuted ordering of >> pages, versus reverse ordering, versus normal ordering. Only the > permuted >> ordering showed the problem. We assume that the permute() function has > an >> unfortunate interaction with the memory allocator in certain HVM guest > OSes, >> causing poor cache utilisation. > > It''s still very odd that the permutation fn only seems to effect Linux > running as a HVM guest and not as a PV guest. I still think there''s > something we''re not quite understanding. > > Jean: have you definitely verified that building a domain with the > permute function does not affect Linux PV guests? >Here a new version of the permute patch, it has to be applied instead of the previous one. Now it works with PV guests, sorry for the delay. Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com> -- Jean Guyader _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel