Xen devel - Jun 2009 - implementing memory ballooning in WIndows

Do the Citrix Windows PV drivers implement memory ballooning? I''ve been
looking at this a bit for GPLPV and the avenues I can see are:

1. Memory can be added via ACPI hot-add, but I assume that that isn''t
implemented in the GPL qemu code (or is it?) and only works for Windows
Server Enterprise, not Vista, XP, or Server Standard.
2. I can give memory back via just allocating a physical page and giving
it back to Xen

Assuming that #1 is undesirable for the reasons stated above, does
anyone have any starting point? Is there a document describing how
dynamic memory works with HVM domains?

Thanks

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Do the Citrix Windows PV drivers implement memory ballooning? I''ve
been
> looking at this a bit for GPLPV and the avenues I can see are:
> 
> 1. Memory can be added via ACPI hot-add, but I assume that that
isn''t
> implemented in the GPL qemu code (or is it?) and only works for Windows
> Server Enterprise, not Vista, XP, or Server Standard.
> 2. I can give memory back via just allocating a physical page and giving
> it back to Xen
Our drivers do implement ballooning (although it''s not actually hooked
up to the control tools, for mostly tedious reasons), and they do it
using approach 2.

Approach 1 does have the advantage that you can later increase a
domain''s memory allocation beyond its boot-time size, but
that''s
obviated to some extent by the new populate-on-demand support, and, as
you say, memory hot add is quite heavily restricted by Microsoft''s
licensing.
> Assuming that #1 is undesirable for the reasons stated above, does
> anyone have any starting point? Is there a document describing how
> dynamic memory works with HVM domains?
Alas no.  The Linux balloon driver has been sighted working in an
unmodified driver build in the past, so that might be worth
investigating, but I couldn''t tell you whether it''s working
now.

This isn''t actually particularly difficult; you pretty much just want
to grab some pages from Windows and then feed them to Xen using
HYPERVISOR_memory_op(XENMEM_decrease_reservation).

Steven.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> > Do the Citrix Windows PV drivers implement memory ballooning?
I''ve
been
> > looking at this a bit for GPLPV and the avenues I can see are:
> >
> > 1. Memory can be added via ACPI hot-add, but I assume that that
isn''t
> > implemented in the GPL qemu code (or is it?) and only works for
Windows
> > Server Enterprise, not Vista, XP, or Server Standard.
> > 2. I can give memory back via just allocating a physical page and
giving
> > it back to Xen
> 
> Our drivers do implement ballooning (although it''s not actually
hooked
> up to the control tools, for mostly tedious reasons), and they do it
> using approach 2.
> 
> Approach 1 does have the advantage that you can later increase a
> domain''s memory allocation beyond its boot-time size, but
that''s
> obviated to some extent by the new populate-on-demand support, and, as
> you say, memory hot add is quite heavily restricted by Microsoft''s
> licensing.
> 
#2 is pretty much where I was going. My concern was that Windows says "I
have 4GB of memory, so I can use X amount for this and Y amount for that
and Z amount for something else", and when the memory is reduced down to
(say) 1GB, I was worried that Windows might get a bit unbalanced. Same
with Linux when it makes decisions about how much space to reserve for
network buffers etc.

But it seems that you have taken the stance that there is no such
problem, so I''ll do the same.

The other thing I was concerned about is that the memory is only given
back once the PV drivers start, so in the situation where you have:
. 16G physical memory
. Dom0 with 2G of memory
. 6 DomU''s with 4G of memory, ballooned down to 2G of memory

It would only work if all the DomU''s successfully ran their PV drivers
and ballooned the memory down as requested, and you couldn''t start them
all at once as on boot they''d actually need 4G of memory.

Did you ever have a tinker with the (mostly) undocumented
MmAddPhysicalMemory function at all? On face value it seems like it is
the sort of function that could be useful... Microsoft would not approve
of such a thing obviously which makes it unusable for production use,
but I am curious as to if it would actually work.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 03/06/2009 15:14, "James Harper"
<james.harper@bendigoit.com.au> wrote:
> #2 is pretty much where I was going. My concern was that Windows says
"I
> have 4GB of memory, so I can use X amount for this and Y amount for that
> and Z amount for something else", and when the memory is reduced down
to
> (say) 1GB, I was worried that Windows might get a bit unbalanced. Same
> with Linux when it makes decisions about how much space to reserve for
> network buffers etc.
> 
> But it seems that you have taken the stance that there is no such
> problem, so I''ll do the same.
It simply restricts how aggressively you can balloon. Ballooning to 1/4 of
original memory, say, will be okay. Ballooning to 1/40 not so much.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > > Do the Citrix Windows PV drivers implement memory ballooning?
I''ve
> been
> > > looking at this a bit for GPLPV and the avenues I can see are:
> > >
> > > 1. Memory can be added via ACPI hot-add, but I assume that that
> isn''t
> > > implemented in the GPL qemu code (or is it?) and only works for
> Windows
> > > Server Enterprise, not Vista, XP, or Server Standard.
> > > 2. I can give memory back via just allocating a physical page and
> giving
> > > it back to Xen
> > 
> > Our drivers do implement ballooning (although it''s not
actually hooked
> > up to the control tools, for mostly tedious reasons), and they do it
> > using approach 2.
> > 
> > Approach 1 does have the advantage that you can later increase a
> > domain''s memory allocation beyond its boot-time size, but
that''s
> > obviated to some extent by the new populate-on-demand support, and, as
> > you say, memory hot add is quite heavily restricted by
Microsoft''s
> > licensing.
> > 
> 
> #2 is pretty much where I was going. My concern was that Windows says
"I
> have 4GB of memory, so I can use X amount for this and Y amount for that
> and Z amount for something else", and when the memory is reduced down
to
> (say) 1GB, I was worried that Windows might get a bit unbalanced. Same
> with Linux when it makes decisions about how much space to reserve for
> network buffers etc.
> 
> But it seems that you have taken the stance that there is no such
> problem, so I''ll do the same.
We''ve gotten away with it so far, certainly.
> The other thing I was concerned about is that the memory is only given
> back once the PV drivers start, so in the situation where you have:
> . 16G physical memory
> . Dom0 with 2G of memory
> . 6 DomU''s with 4G of memory, ballooned down to 2G of memory
> 
> It would only work if all the DomU''s successfully ran their PV
drivers
> and ballooned the memory down as requested, and you couldn''t start
them
> all at once as on boot they''d actually need 4G of memory.
Well, the second problem is solved by populate-on-demand mode.  The
first is much harder, and almost certainly requires some kind of
emergency hypervisor paging.

Of course, if you do populate-on-demand and then *can''t* balloon down
then the failure mode is extremely bad.  It can hopefully be avoided
by a sufficiently paranoid toolstack, but it''s still rather
unfortunate.
> Did you ever have a tinker with the (mostly) undocumented
> MmAddPhysicalMemory function at all? On face value it seems like it is
> the sort of function that could be useful... Microsoft would not approve
> of such a thing obviously which makes it unusable for production use,
> but I am curious as to if it would actually work.
I''ve not looked at it at all, sorry.

Steven.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jun 3, 2009 at 2:40 PM, Steven Smith <steven.smith@citrix.com>
wrote:
> Alas no.  The Linux balloon driver has been sighted working in an
> unmodified driver build in the past, so that might be worth
> investigating, but I couldn''t tell you whether it''s
working now.
It was working when I was testing the populate-on-demand functionality
back in December / January timeframe.  I used Linux running in HVM
mode with xen_balloon drivers exclusively for my testing.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

James --

I''m completely ignorant about Windows but have often wondered
if self-ballooning scripts (which today only run in a RedHat-ish
environment) could be implemented on Windows.

If you might be interested, look in the tree (starting
around Xen 3.3) at tools/xenballoon, with more info
on the Xen Summit Summer 2008 site.

Dan
> -----Original Message-----
> From: Steven Smith [mailto:steven.smith@eu.citrix.com]
> Sent: Wednesday, June 03, 2009 8:58 AM
> To: James Harper
> Cc: Steven Smith; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] implementing memory ballooning in WIndows
> 
> 
> > > > Do the Citrix Windows PV drivers implement memory 
> ballooning? I''ve
> > been
> > > > looking at this a bit for GPLPV and the avenues I can see
are:
> > > >
> > > > 1. Memory can be added via ACPI hot-add, but I assume that
that
> > isn''t
> > > > implemented in the GPL qemu code (or is it?) and only works
for
> > Windows
> > > > Server Enterprise, not Vista, XP, or Server Standard.
> > > > 2. I can give memory back via just allocating a 
> physical page and
> > giving
> > > > it back to Xen
> > > 
> > > Our drivers do implement ballooning (although it''s not 
> actually hooked
> > > up to the control tools, for mostly tedious reasons), and 
> they do it
> > > using approach 2.
> > > 
> > > Approach 1 does have the advantage that you can later increase a
> > > domain''s memory allocation beyond its boot-time size,
but that''s
> > > obviated to some extent by the new populate-on-demand 
> support, and, as
> > > you say, memory hot add is quite heavily restricted by
Microsoft''s
> > > licensing.
> > > 
> > 
> > #2 is pretty much where I was going. My concern was that 
> Windows says "I
> > have 4GB of memory, so I can use X amount for this and Y 
> amount for that
> > and Z amount for something else", and when the memory is 
> reduced down to
> > (say) 1GB, I was worried that Windows might get a bit 
> unbalanced. Same
> > with Linux when it makes decisions about how much space to 
> reserve for
> > network buffers etc.
> > 
> > But it seems that you have taken the stance that there is no such
> > problem, so I''ll do the same.
> We''ve gotten away with it so far, certainly.
> 
> > The other thing I was concerned about is that the memory is 
> only given
> > back once the PV drivers start, so in the situation where you have:
> > . 16G physical memory
> > . Dom0 with 2G of memory
> > . 6 DomU''s with 4G of memory, ballooned down to 2G of memory
> > 
> > It would only work if all the DomU''s successfully ran their 
> PV drivers
> > and ballooned the memory down as requested, and you 
> couldn''t start them
> > all at once as on boot they''d actually need 4G of memory.
> Well, the second problem is solved by populate-on-demand mode.  The
> first is much harder, and almost certainly requires some kind of
> emergency hypervisor paging.
> 
> Of course, if you do populate-on-demand and then *can''t* balloon
down
> then the failure mode is extremely bad.  It can hopefully be avoided
> by a sufficiently paranoid toolstack, but it''s still rather
> unfortunate.
> 
> > Did you ever have a tinker with the (mostly) undocumented
> > MmAddPhysicalMemory function at all? On face value it seems 
> like it is
> > the sort of function that could be useful... Microsoft 
> would not approve
> > of such a thing obviously which makes it unusable for 
> production use,
> > but I am curious as to if it would actually work.
> I''ve not looked at it at all, sorry.
> 
> Steven.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel