Xen devel - May 2006 - [PATCH][BALLOON] Fix minimum target

Keir,

We tried a few experiments with your 2% patch. We found that 2% is not
good enough. The magic numbers seem to be 3% for DomUs and 5% for Dom0.
Any idea why the difference in percentages?

If this patch is fine by you, could you please apply to both
xen-unstable and xen-3.0-testing?

Thanks,
Aravindh Puthiyaparambil 
Xen Development Team
Unisys, Tredyffrin PA

--------------------------

Do not allow target to be set below 3% of maximum memory for DomUs and
5% for Dom0.
Signed-off-by: Aravindh Puthiyaparambil
<aravindh.puthiyaparambil@unisys.com>




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27 May 2006, at 06:18, Puthiyaparambil, Aravindh wrote:
> We tried a few experiments with your 2% patch. We found that 2% is not
> good enough. The magic numbers seem to be 3% for DomUs and 5% for Dom0.
> Any idea why the difference in percentages?
The minimum should probably include a fixed-size base value, plus some 
percentage of maximum possible memory size. The patch I applied does 
not include the fixed-size base value as it isn''t obvious what to set 
it to.

The percentage you need (5%) for domain0 may appear bigger for a few 
reasons:
  * There''s a bigger static allocation of memory in domain0,
independent
of maximum memory size
  * You allocate less initial memory to domain0, so the fixed-size 
reserved memory is a bigger proprtion of total reserved memory.

What happens if you set the min to 16MB + 2% of max-mem?

  -- Keir
> If this patch is fine by you, could you please apply to both
> xen-unstable and xen-3.0-testing?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> What happens if you set the min to 16MB + 2% of max-mem?
I changed the balloon driver to reflect this and found that DomU freezes
when it reaches around 2.2% - 3.1% of it initial allocation. I have the
data below for 2GB, 4GB, 8GB and 12GB DomUs. The MEM(k) below shows info
from xentop (with 1s delay) when the DomUs froze. Initial Alloc (%)is
what I calculated by hand. I am not sure how accurate xentop is so
Initial Alloc % could be a little lower than what is shown.

NAME        MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) Initial Alloc(%)
2GBmem      66560    0.4    2101248      12.6   3.1 (xm mem-set id 32M)
4GBmem     113664    0.7    4198400      25.2   2.7 (xm mem-set id 64M)
8GBmem     205824    1.2    8392704      50.4   2.4 (xm mem-set id 64M)
12GBmem    281600    1.7   12587008      75.6   2.2 (xm mem-set id 64M)

So it looks like initial alloc % grows as domains get smaller. It does
not make sense. I think I am missing something here. Do you have any
clues? This could very well be the DomU crashing before xentop can
update the info.

I ran a few tests and it seems that setting the min to 64MB + 2% of
max-mem seems like a good option for DomUs.

Dom0 needs a much bigger floor of 192M. I think this is where KY came up
with the 192M number. I know this will cause problems on machines coming
up with dom0_mem<192M. 

Both these options will break xm-test. So how do you want to proceed?

Cheers,
Aravindh	

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27 May 2006, at 21:34, Puthiyaparambil, Aravindh wrote:
> Dom0 needs a much bigger floor of 192M. I think this is where KY came 
> up
> with the 192M number. I know this will cause problems on machines 
> coming
> up with dom0_mem<192M.
That being the case (which it certainly is) it is pretty obvious that a 
floor of 192M is not suitable in all situations.
> Both these options will break xm-test. So how do you want to proceed?
If there isn''t a suitable static minimum which prevents OOM death on 
most systems without also impacting the useful lower range of 
ballooning on other systems then I think we''ll have to wait for someone
to implement a more dynamic scheme (e.g., by hooking off the 
OOM/low-memory paths, or by slowly allocating memory only when we can 
see a reasonable amount of pages available on the free lists).

I''m rather doubtful that a really good static estimate can be derived, 
since it depends so much on particular details of kernel memory usage. 
Given the results of these experiments I''m tempted to remove the 2% 
minimum that is already in the tree -- vendors can make their own patch 
if they have a better idea of what works for their particular kernel 
setups.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> ballooning on other systems then I think we''ll have to wait for
someone
> to implement a more dynamic scheme (e.g., by hooking off the
> OOM/low-memory paths, or by slowly allocating memory only when we can
> see a reasonable amount of pages available on the free lists).
>From the little code reading I have done, your second option seems
better. Once the code goes down the OOM path the system is already in a
shaky unstable state. The balloon driver has to be made to back off way
before that. I have been meaning to try and look into this. I will try
and do so. 
> I''m rather doubtful that a really good static estimate can be
derived,
> since it depends so much on particular details of kernel memory usage.
I agree.
> Given the results of these experiments I''m tempted to remove the
2%
> minimum that is already in the tree -- vendors can make their own
patch
> if they have a better idea of what works for their particular kernel
> setups.
I second this opinion. I think it is better if you remove the 2%
minimum. It really serves no purpose in the majority of situations.

Aravindh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 May 2006, at 03:01, Puthiyaparambil, Aravindh wrote:
>> ballooning on other systems then I think we''ll have to wait
for
> someone
>> to implement a more dynamic scheme (e.g., by hooking off the
>> OOM/low-memory paths, or by slowly allocating memory only when we can
>> see a reasonable amount of pages available on the free lists).
>
> From the little code reading I have done, your second option seems
> better. Once the code goes down the OOM path the system is already in a
> shaky unstable state. The balloon driver has to be made to back off way
> before that. I have been meaning to try and look into this. I will try
> and do so.
The two strategies I suggested could broadly be described as 
event-driven quench (kick the balloon driver off some existing or 
modified path in the mm code) or polling based (balloon driver 
periodically evaluates the status of the memory system for itself). 
Either way, the key to getting this to work properly will be to find a 
simple but effective metric for evaluating memory pressure. Something 
that does not stop the balloon driver, or slow it down too much, until 
it gets close to the crunch, but is basically 100% guaranteed to kick 
in before oom.

We do need to implement this without slowing down the balloon driver 
too much. Slowing it down by more than a factor of two, say, might well 
be unacceptable. Even then there will likely be timeouts in xend and 
xm-test that will need adjusting. Still, I think this is quite a 
necessary feature so I suppose we''ll see how it turns out when it is 
implemented. I agree that trying out the ''measurement from within the 
balloon driver'' approach first makes sense.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The idea was to have in place a structure that allowed for some
experimentation on exactly what the floor needs to be and what the
minimum memory slope needs to be.  An earlier version of my patch
attempted to classify machines based on the maximum amount of memory
that it would ever handle. For each class the minimum memory was still a
linear function with the slope and the floor being dependent on the
maximum amount of memory that the domain would handle. Another approach
I was considering was to base the amount of minimum memory on the amount
of free memory a domain has. This strategy may give us a reasonably
functional system when the domain is ballooned down while it may result
in a minimum that could be potentially higher than an approach that we
have been playing with might yield.

Regards,

K. Y
>>> On Sat, May 27, 2006 at  6:46 pm, in message
<fe38b039cfc2775cdaa555c35a441c94@cl.cam.ac.uk>, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote: 
> On 27 May 2006, at 21:34, Puthiyaparambil, Aravindh wrote:
> 
>> Dom0 needs a much bigger floor of 192M. I think this is where KY
came 
>> up
>> with the 192M number. I know this will cause problems on machines 
>> coming
>> up with dom0_mem<192M.
> 
> That being the case (which it certainly is) it is pretty obvious that
a 
> floor of 192M is not suitable in all situations.
> 
>> Both these options will break xm- test. So how do you want to
proceed?
> 
> If there isn''t a suitable static minimum which prevents OOM death
on
> most systems without also impacting the useful lower range of 
> ballooning on other systems then I think we''ll have to wait for
someone 
> to implement a more dynamic scheme (e.g., by hooking off the 
> OOM/low- memory paths, or by slowly allocating memory only when we
can 
> see a reasonable amount of pages available on the free lists).
> 
> I''m rather doubtful that a really good static estimate can be
derived, 
> since it depends so much on particular details of kernel memory
usage. 
> Given the results of these experiments I''m tempted to remove the
2%
> minimum that is already in the tree --  vendors can make their own
patch 
> if they have a better idea of what works for their particular kernel
> setups.
> 
>   --  Keir
> 
> 
> _______________________________________________
> Xen- devel mailing list
> Xen- devel@lists.xensource.com
> http://lists.xensource.com/xen- devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The idea was to have in place a structure that allowed for some
> experimentation on exactly what the floor needs to be and what the
> minimum memory slope needs to be.  An earlier version of my patch
> attempted to classify machines based on the maximum amount of memory
> that it would ever handle. For each class the minimum memory was still a
> linear function with the slope and the floor being dependent on the
> maximum amount of memory that the domain would handle. Another approach
> I was considering was to base the amount of minimum memory on the amount
> of free memory a domain has. This strategy may give us a reasonably
> functional system when the domain is ballooned down while it may result
> in a minimum that could be potentially higher than an approach that we
> have been playing with might yield.
I don''t believe any static predetermined floor is going to work
well. Basically I expect that any such approach will either:
 A. Exhibit many false negatives (allow ballooning beyond the critical
 point and result in OOM)
 B. Exhibit many false positives (disallow ballooning when it would
 actually be safe)
 C. Be an unmaintainable mess of special cases derived via empirical
 analysis that will not scale or continue to work well moving forward

I''m particularly bothered by B as it is harder to prove that a
particular patch has that problem. But for example, I''m sure a static
floor of 192MB falls squarely into B.

Unless someone convinces me otherwise, I hold out more hope for the
kind of dynamic approach I''ve been discussing with Aravindh.

 -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel