George Dunlap
2011-Mar-09 13:47 UTC
[Xen-devel] RFC: Doing a superpage zero-sweep on decrease_reservation
I''m going through the populate-on-demand patches that we''ve had in our XenServer patchqueue and trying to figure out what he best thing is to do. One of the things we implemented was a hypercall that would allow the tools to initiate a zero-page sweep. The idea for this actually was to try to help keep contiguous superpages if possible. Asking Windows for 2meg pages couldn''t be made to work properly (Paul said, you can ask Windows, but it just wouldn''t give any). When ballooning down, the balloon driver first calls for a zero-page sweep, hoping Xen will find big contiguous zero superpages it can reclaim; then as the balloon driver returns individual pages, eventually it will end up filling in a lot of whole superpages anyway. The problem with this is that for machines with very large memory, this hypercall can take a long time; and if the VM has already been running for a while, it''s likely that a significant portion of the space *won''t* be zero. It occured to me, it might be simpler if we check for a zero superpage when doing a decrease_reservation call instead. We''d only do a sweep if the p2m entry is currently a superpage. If that''s the case, we sweep the superpage containing that mfn and nothing else. If the whole superpage isn''t zero, we''ll shatter the superpage in the p2m tables, so that even if we get an mfn from the same page again, we won''t scan it. I was going to say, "scan only if the number of p2m entries is higher than the number of entries in the cache". But it occurred to me, it might not be a bad idea to scan for zero pages in any case -- so that we can consolidate even after the guest has been running for a while. Thoughts? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Mar-09 14:23 UTC
[Xen-devel] Re: RFC: Doing a superpage zero-sweep on decrease_reservation
Hi, At 13:47 +0000 on 09 Mar (1299678468), George Dunlap wrote:> It occured to me, it might be simpler if we check for a zero superpage > when doing a decrease_reservation call instead. We''d only do a sweep > if the p2m entry is currently a superpage. If that''s the case, we > sweep the superpage containing that mfn and nothing else. If the > whole superpage isn''t zero, we''ll shatter the superpage in the p2m > tables, so that even if we get an mfn from the same page again, we > won''t scan it.Do you think ballooned-out pages are likely to be in 2MB chunks of zeroed memory? Apart from avoiding shattering and re-gathering a 2MB p2m range (which is nice, but proabbly not the perf bottleneck for PoD right now), this is a change from scanning arbitrary memory when we run out to scanning memory around balloon sites as we go. If this is likely to have a high hit rate, great. If not, it might be (overall) worse because we waste time scanning as we balloon and have to do the full scan later anyway.> I was going to say, "scan only if the number of p2m entries is higher > than the number of entries in the cache". But it occurred to me, it > might not be a bad idea to scan for zero pages in any case -- so that > we can consolidate even after the guest has been running for a while.I''m not so keen on that - AFAICS once we''ve passed #pod-entries =#pod-pages, we shouldn''t be doing any expensive work for PoD. We might want to think about defragging HVM guests independently of the PoD stuff, though. Cheers, Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2011-Mar-09 14:46 UTC
[Xen-devel] Re: RFC: Doing a superpage zero-sweep on decrease_reservation
On Wed, 2011-03-09 at 14:23 +0000, Tim Deegan wrote:> Hi, > > At 13:47 +0000 on 09 Mar (1299678468), George Dunlap wrote: > > It occured to me, it might be simpler if we check for a zero superpage > > when doing a decrease_reservation call instead. We''d only do a sweep > > if the p2m entry is currently a superpage. If that''s the case, we > > sweep the superpage containing that mfn and nothing else. If the > > whole superpage isn''t zero, we''ll shatter the superpage in the p2m > > tables, so that even if we get an mfn from the same page again, we > > won''t scan it. > > Do you think ballooned-out pages are likely to be in 2MB chunks of > zeroed memory? Apart from avoiding shattering and re-gathering a 2MB > p2m range (which is nice, but proabbly not the perf bottleneck for PoD > right now), this is a change from scanning arbitrary memory when we run > out to scanning memory around balloon sites as we go.On the contrary, the function of the hypercall was to do a full memory sweep regardless of whether the cache was running low or not -- that is, the purpose of the hypercall (as used by the Citrix PV drivers) has nothing to do with out-of-memory situations; it only has to do with superpage consolidation. (Avoiding confusion in the future over what the hypercall is actually for is another good reason not to re-use the emergency sweep code if we don''t need to!)> If this is likely to have a high hit rate, great. If not, it might be > (overall) worse because we waste time scanning as we balloon and have to > do the full scan later anyway.This shouldn''t have a significant impact on the number of emergency sweeps we do -- in theory we shouldn''t need to do any sweeps once the balloon driver starts handing memory back to Xen. But I will run some tests, to see what impact it has on boot time, and number of intact superpages returned to Xen as a part of the initial balloon process.> > I was going to say, "scan only if the number of p2m entries is higher > > than the number of entries in the cache". But it occurred to me, it > > might not be a bad idea to scan for zero pages in any case -- so that > > we can consolidate even after the guest has been running for a while. > > I''m not so keen on that - AFAICS once we''ve passed #pod-entries => #pod-pages, we shouldn''t be doing any expensive work for PoD. We might > want to think about defragging HVM guests independently of the PoD > stuff, though.That sounds reasonable. OK, hopefully I''ll have a patch (with some experiments to back it up) in a week or so. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2011-Mar-09 15:00 UTC
[Xen-devel] Re: RFC: Doing a superpage zero-sweep on decrease_reservation
On Wed, 2011-03-09 at 14:23 +0000, Tim Deegan wrote:> Do you think ballooned-out pages are likely to be in 2MB chunks of > zeroed memory?Sorry, didn''t answer this. I think that for a large guest, the initial balloon is very likely that for the first mfn freed in a superpage, that mfn will turn out to have been in a 2MB chunks of zeroed memory, all of which will eventually passed back to Xen anyway. (e.g., imagine a VM with 32GiB static_max but a 16GiB target -- of the 16GiB of individual 4k pages passed back by the balloon driver, the vast majority of them will be in a page which (1) is zeroed already and (2) will end up being passed back entirely anyway.) One could imagine the PV drivers allocating the full balloon size, then going through and sorting the pages by pfn, consolidating the superpages, and passing them back as one big chunk -- but that would be a big waste of time compared to just having Xen optimistically reclaim the whole superpage. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel