How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral store? In my use case, I would prefer that an old page was automatically discarded to make room for a new page, and that pages that have been retrieved already are more likely to be discarded than a page that has not yet been retrieved. Thanks James
On Sun, Jun 02, 2013 at 04:06:18AM +0000, James Harper wrote:> How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral store? In my use case, I would prefer that an old page was automatically discarded to make room for a new page, and that pages that have been retrieved already are more likely to be discarded than a page that has not yet been retrieved. >In other words an LRU system with bounds. Right now the bounds are the amount of memory available and when Xen needs more memory the bounds gets decreased - which will discard the least used ephermeal pages. There is a tmem subop to actually set the cap (TMEMC_SET_CAP), which sets the cap value. However I am not seeing in the code anybody using it. Hmmm> Thanks > > James > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
> From: Konrad Rzeszutek Wilk > Sent: Monday, June 03, 2013 6:22 AM > To: James Harper > Cc: xen-devel@lists.xen.org; Dan Magenheimer > Subject: Re: [Xen-devel] tmem ephemeral page discarding > > On Sun, Jun 02, 2013 at 04:06:18AM +0000, James Harper wrote: > > How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral > store? In my use case, I would prefer that an old page was automatically discarded to make room for a > new page, and that pages that have been retrieved already are more likely to be discarded than a page > that has not yet been retrieved. > > In other words an LRU system with bounds. Right now the bounds are the amount of memory available and > when Xen needs more memory the bounds gets decreased - which will discard the least used ephermeal > pages.It sounds like Xen tmem matches your use case. Ephemeral pages that have been retrieved are actually removed from tmem, i.e. ephemeral gets are "destructive", i.e. an ephemeral get implies a flush. (Note: This isn''t true for shared ephemeral pages, but I don''t think you care about clustered filesystems so ignore that case for now.) And ephemeral pages (as Konrad replied) are kept in a LRU queue across all (tmem-aware) guests and the LRU ephemeral page across all guests is discarded. Or if a weight limit is exceeded, the LRU ephemeral page belonging to the current guest is discarded.> There is a tmem subop to actually set the cap (TMEMC_SET_CAP), which sets the cap value. However > I am not seeing in the code anybody using it. HmmmSee discussion in "Weights and caps" in docs/misc/tmem-internals.html (or on the web at https://oss.oracle.com/projects/tmem/dist/documentation/internals/xen4-internals-v01.html ) The current cap non-implementation is per-guest, but I never found a good use model for it so never implemented it, outside of the toolstack control interface. Caps exist only because "weights+caps" make sense for a CPU scheduler and I thought both might make sense also for a "memory scheduler". If you have a use model for it (or a use model for across-all-guests cap), it shouldn''t be hard to implement. Hope that helps! Dan
> > And ephemeral pages (as Konrad replied) are kept in a LRU queue across all > (tmem-aware) guests and the LRU ephemeral page across all guests is > discarded. Or if a weight limit is exceeded, the LRU ephemeral > page belonging to the current guest is discarded. >Yep that sounds about perfect. I can''t seem to get it working under 64 bit windows at the moment... keeps returning -ENOSYS and I''m wondering if windows and linux differ in their structure alignment or something. I remember this being a problem with vbd. Btw, the choice of ''new'' as a struct name in the tmem struct isn''t necessarily wise... it might never come up but ''new'' is a reserved word in c++. James
> From: James Harper [mailto:james.harper@bendigoit.com.au] > Sent: Monday, June 03, 2013 7:42 PM > To: Dan Magenheimer; Konrad Wilk > Cc: xen-devel@lists.xen.org > Subject: RE: [Xen-devel] tmem ephemeral page discarding > > > > > And ephemeral pages (as Konrad replied) are kept in a LRU queue across all > > (tmem-aware) guests and the LRU ephemeral page across all guests is > > discarded. Or if a weight limit is exceeded, the LRU ephemeral > > page belonging to the current guest is discarded. > > > > Yep that sounds about perfect. > > I can''t seem to get it working under 64 bit windows at the moment... keeps returning -ENOSYS and I''m > wondering if windows and linux differ in their structure alignment or something. I remember this being > a problem with vbd.Any chance you''ve not enabled it? Tmem is disabled by default and you need to provide "tmem" as a boot parameter to xen to enable it. Tmem calls from a guest will fail if it is disabled.> Btw, the choice of ''new'' as a struct name in the tmem struct isn''t necessarily wise... it might never > come up but ''new'' is a reserved word in c++.Hmmm... Where are you seeing it? I do see it in the tmem code as a local parameter ("new" is used in code in other places in Xen as well), but not in a public structure definition. Thanks, Dan
> > I can''t seem to get it working under 64 bit windows at the moment... keeps > > returning -ENOSYS and I''m > > wondering if windows and linux differ in their structure alignment or > > something. I remember this being > > a problem with vbd. > > Any chance you''ve not enabled it? Tmem is disabled by default > and you need to provide "tmem" as a boot parameter to xen to > enable it. Tmem calls from a guest will fail if it is > disabled.I botched a cut&paste when adding the new x64 hypercall to GPLPV. Oops. All good now> > Btw, the choice of ''new'' as a struct name in the tmem struct isn''t > necessarily wise... it might never > > come up but ''new'' is a reserved word in c++. > > Hmmm... Where are you seeing it? I do see it in the tmem > code as a local parameter ("new" is used in code in other > places in Xen as well), but not in a public structure definition. >I was looking at very old source and now I see it was resolved over 2 years ago http://xenbits.xen.org/hg/xen-unstable.hg/rev/0353037c6b95 Thanks James