Xen devel - May 2006 - [PATCH] Fix auto-ballooning of dom0 for HVM domains

Hi,

I''ve been trying to make the auto-ballooning of domain 0 work reliably
for HVM domains.

Patch #1 is a simple bug fix, in preparation for patch #2.  Patch #2
tries to calculate how much memory is needed for HVM domains (XenSource
bug #521) and is certainly open for discussion.  Patches apply cleanly
to both 3.0-testing and unstable.

Patch #1 (xen-hvm-auto-balloon.diff):
When a domain (whether para- or fully-virtualized) reports how much
overhead memory it requires (via getDomainMemory in image.py), all such
memory was immediately allocated to the domain itself.  This is
certainly incorrect for HVM domains, since additional
increase_reservation calls are made later in qemu.  Since all ballooned
memory is already taken, qemu will fail.  The fix is to treat the
requested memory size and the overhead size as separate values.  The
requested memory size is immediately allocated to the new domain; the
overhead is left unallocated for whatever else might need it later.

Patch #2 (xen-get-dom-memory.diff):
This patch calculates the overhead needed for HVM domains.  If HVM is
supported by the hardware, I add a little ballooning overhead to
paravirtualized VMs also, to avoid low-memory situations.  (There are
various unchecked alloc_domheap_pages calls in shadow*.c that I am
trying to avoid tripping over for now...)  The values in this patch work
fine on 32 bit; I may update them later based on feedback and/or testing
on 64 bit.

Thanks,
Chuck




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sorry, I forgot the Signed-off-by line.  Consider this my sign-off on
the previous email:

Signed-off-by: Charles Coffing <ccoffing@novell.com>

>>> On Thu, May 18, 2006 at 11:04 AM, in message
<446C5554.D169.003C.0@novell.com>,
"Charles Coffing" <ccoffing@novell.com> wrote:
> Hi,
> 
> I''ve been trying to make the auto- ballooning of domain 0 work
reliably
> for HVM domains.
> 
> Patch #1 is a simple bug fix, in preparation for patch #2.  Patch #2
> tries to calculate how much memory is needed for HVM domains
(XenSource
> bug #521) and is certainly open for discussion.  Patches apply
cleanly
> to both 3.0- testing and unstable.
> 
> Patch #1 (xen- hvm- auto- balloon.diff):
> When a domain (whether para-  or fully- virtualized) reports how
much
> overhead memory it requires (via getDomainMemory in image.py), all
such
> memory was immediately allocated to the domain itself.  This is
> certainly incorrect for HVM domains, since additional
> increase_reservation calls are made later in qemu.  Since all
ballooned
> memory is already taken, qemu will fail.  The fix is to treat the
> requested memory size and the overhead size as separate values.  The
> requested memory size is immediately allocated to the new domain;
the
> overhead is left unallocated for whatever else might need it later.
> 
> Patch #2 (xen- get- dom- memory.diff):
> This patch calculates the overhead needed for HVM domains.  If HVM
is
> supported by the hardware, I add a little ballooning overhead to
> paravirtualized VMs also, to avoid low- memory situations.  (There
are
> various unchecked alloc_domheap_pages calls in shadow*.c that I am
> trying to avoid tripping over for now...)  The values in this patch
work
> fine on 32 bit; I may update them later based on feedback and/or
testing
> on 64 bit.
> 
> Thanks,
> Chuck


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Charles
	Just one suggestion, for  xen-hvm-auto-balloon.diff, how about
change 
             xc.domain_setmaxmem(self.domid, m)
 
to 
             xc.domain_setmaxmem(self.domid,
self.info[''memory''] * 1024)

The maxmem would be used to check when guest try to increase memory.

Also, does anyone know the what the BALLOON_OUT_SLACK on
tools/python/xen/xend/balloon.py stands for? The comments said it is
"physinfo details are rounded".

Thanks
Yunhong Jiang

 >-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com 
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
>Charles Coffing
>Sent: Friday, May 19, 2006 1:04 AM
>To: xen-devel@lists.xensource.com
>Subject: [Xen-devel] [PATCH] Fix auto-ballooning of dom0 for 
>HVM domains
>
>Hi,
>
>I''ve been trying to make the auto-ballooning of domain 0 work
reliably
>for HVM domains.
>
>Patch #1 is a simple bug fix, in preparation for patch #2.  Patch #2
>tries to calculate how much memory is needed for HVM domains (XenSource
>bug #521) and is certainly open for discussion.  Patches apply cleanly
>to both 3.0-testing and unstable.
>
>Patch #1 (xen-hvm-auto-balloon.diff):
>When a domain (whether para- or fully-virtualized) reports how much
>overhead memory it requires (via getDomainMemory in image.py), all such
>memory was immediately allocated to the domain itself.  This is
>certainly incorrect for HVM domains, since additional
>increase_reservation calls are made later in qemu.  Since all ballooned
>memory is already taken, qemu will fail.  The fix is to treat the
>requested memory size and the overhead size as separate values.  The
>requested memory size is immediately allocated to the new domain; the
>overhead is left unallocated for whatever else might need it later.
>
>Patch #2 (xen-get-dom-memory.diff):
>This patch calculates the overhead needed for HVM domains.  If HVM is
>supported by the hardware, I add a little ballooning overhead to
>paravirtualized VMs also, to avoid low-memory situations.  (There are
>various unchecked alloc_domheap_pages calls in shadow*.c that I am
>trying to avoid tripping over for now...)  The values in this 
>patch work
>fine on 32 bit; I may update them later based on feedback 
>and/or testing
>on 64 bit.
>
>Thanks,
>Chuck
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, May 19, 2006 at 12:23:59PM +0800, Jiang, Yunhong wrote:
> Hi, Charles
> 	Just one suggestion, for  xen-hvm-auto-balloon.diff, how about
> change 
>              xc.domain_setmaxmem(self.domid, m)
>  
> to 
>              xc.domain_setmaxmem(self.domid,
self.info[''memory''] * 1024)
> 
> The maxmem would be used to check when guest try to increase memory.
> 
> Also, does anyone know the what the BALLOON_OUT_SLACK on
> tools/python/xen/xend/balloon.py stands for? The comments said it is
> "physinfo details are rounded".
When you get the free memory from xc.physinfo(), it may report (for example)
100MB free when actually there are 99.9MB free.  It rounds up because you
are often just a few pages short of the value you expect, because there are
pages being used for the I/O shared rings, for example.  This means that when
deciding how much memory to balloon, we need to subtract 1MB.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan
	Thanks for your explaination very much !

Thanks
Yunhong Jiang 
>-----Original Message-----
>From: Ewan Mellor [mailto:ewan@xensource.com] 
>Sent: Friday, May 19, 2006 11:15 PM
>To: Jiang, Yunhong
>Cc: Charles Coffing; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] [PATCH] Fix auto-ballooning of dom0 
>for HVM domains
>
>On Fri, May 19, 2006 at 12:23:59PM +0800, Jiang, Yunhong wrote:
>
>> Hi, Charles
>> 	Just one suggestion, for  xen-hvm-auto-balloon.diff, how about
>> change 
>>              xc.domain_setmaxmem(self.domid, m)
>>  
>> to 
>>              xc.domain_setmaxmem(self.domid, 
>self.info[''memory''] * 1024)
>> 
>> The maxmem would be used to check when guest try to increase memory.
>> 
>> Also, does anyone know the what the BALLOON_OUT_SLACK on
>> tools/python/xen/xend/balloon.py stands for? The comments said it is
>> "physinfo details are rounded".
>
>When you get the free memory from xc.physinfo(), it may report 
>(for example)
>100MB free when actually there are 99.9MB free.  It rounds up 
>because you
>are often just a few pages short of the value you expect, 
>because there are
>pages being used for the I/O shared rings, for example.  This 
>means that when
>deciding how much memory to balloon, we need to subtract 1MB.
>
>Ewan.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> On Thu, May 18, 2006 at 10:23 PM, in message
<A8F9FF3706D1A5479EF62192B976DB441C0594@pdsmsx401.ccr.corp.intel.com>,
"Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:
> Hi, Charles
> 	Just one suggestion, for  xen- hvm- auto- balloon.diff, how
about
> change 
>              xc.domain_setmaxmem(self.domid, m)
>  
> to 
>              xc.domain_setmaxmem(self.domid,
self.info[''memory''] *
1024)

Ideally, yes, I would agree.  But later, in qemu, another
increase_reservation() is called.  If I make the suggested change, I
suspect that qemu will fail to get its memory.

Or is this upper limit not checked when increase_reservation() is
called from dom0?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Charles Coffing
>Sent: 2006年5月20日 3:59
>> On Thu, May 18, 2006 at 10:23 PM, in message
><A8F9FF3706D1A5479EF62192B976DB441C0594@pdsmsx401.ccr.c
>orp.intel.com>,
>"Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:
>> Hi, Charles
>> 	Just one suggestion, for  xen- hvm- auto- balloon.diff, how
>about
>> change
>>              xc.domain_setmaxmem(self.domid, m)
>>
>> to
>>              xc.domain_setmaxmem(self.domid,
self.info[''memory'']
>*
>1024)
>
>Ideally, yes, I would agree.  But later, in qemu, another
>increase_reservation() is called.  If I make the suggested change, I
>suspect that qemu will fail to get its memory.
>
>Or is this upper limit not checked when increase_reservation() is
>called from dom0?
BTW, could you please explain why following change is required:

+        given amount, also in KiB.  This is normally just mem, but if 
HVM is
+        supported, keep a little extra free."""
+        if ''hvm'' in
xc.xeninfo()[''xen_caps'']:
+            mem_kb += 4*1024;
+        return mem_kb

Why do you need to reserve extra memory even for domU as long as 
the platform supports hvm feature? 

P.S I''ll send a patch to reverse this change for ia64, since balloon
doesn''t
work for xen/ia64 yet and thus we have to allocate all memory at creation 
time.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22 May 2006, at 09:51, Tian, Kevin wrote:
> Why do you need to reserve extra memory even for domU as long as
> the platform supports hvm feature?
>
> P.S I''ll send a patch to reverse this change for ia64, since
balloon
> doesn''t
> work for xen/ia64 yet and thus we have to allocate all memory at 
> creation
> time.
It would be nice to avoid needing that kludge at all even on x86. It 
isn''t entirely clear to me why it''s needed. If it''s
causing problems
for ia64 I''m inclined to remove it unless there is a concrete reason 
for keeping it.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2006年5月22日 17:06
>
>
>On 22 May 2006, at 09:51, Tian, Kevin wrote:
>
>> Why do you need to reserve extra memory even for domU as long as
>> the platform supports hvm feature?
>>
>> P.S I''ll send a patch to reverse this change for ia64, since
balloon
>> doesn''t
>> work for xen/ia64 yet and thus we have to allocate all memory at
>> creation
>> time.
>
>It would be nice to avoid needing that kludge at all even on x86. It
>isn''t entirely clear to me why it''s needed. If
it''s causing problems
>for ia64 I''m inclined to remove it unless there is a concrete
reason
>for keeping it.
>
>  -- Keir
Yes, that breaks xen/ia64 for both domU and domVTI. Though I sent 
out a workaround just now, it''s better if you can reverse it directly.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Mon, May 22, 2006 at  2:51 AM, in message
<571ACEFD467F7749BC50E0A98C17CDD8094E7C7E@pdsmsx403>, "Tian,
Kevin"
<kevin.tian@intel.com> wrote: 
> BTW, could you please explain why following change is required:
> 
> +        given amount, also in KiB.  This is normally just mem, but
if 
> HVM is
> +        supported, keep a little extra free."""
> +        if ''hvm'' in
xc.xeninfo()[''xen_caps'']:
> +            mem_kb += 4*1024;
> +        return mem_kb
> 
> Why do you need to reserve extra memory even for domU as long as 
> the platform supports hvm feature? 
HVM domains need some extra memory free for the shadow page tables,
otherwise they may crash the entire machine while they are running, or
the HVM domain itself may crash (exact behavior depends on whether you
have Yunhong''s patch to change BUGs to domain_crash).

This slack space is calculated into the memory size for HVM domains,
but what happens if you then start a PV domain afterwards?  Only the
minimally required memory is freed up, then the PV domain takes it all,
leaving the HVM domain with no slack == crash.

Admittedly, a better solution is to only add on the slack to PV if an
HVM domain is actually running, not merely supported.

BTW, in my testing on 32 bit, a Linux HVM domain consumes an additional
~3 MB over the course of fully booting and starting a few apps.  As I
start more apps, the memory consumption goes up more...



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22 May 2006, at 15:51, Charles Coffing wrote:
> HVM domains need some extra memory free for the shadow page tables,
> otherwise they may crash the entire machine while they are running, or
> the HVM domain itself may crash (exact behavior depends on whether you
> have Yunhong''s patch to change BUGs to domain_crash).
>
> This slack space is calculated into the memory size for HVM domains,
> but what happens if you then start a PV domain afterwards?  Only the
> minimally required memory is freed up, then the PV domain takes it all,
> leaving the HVM domain with no slack == crash.
Where does the slack from the previous HVM guest startup go? Are you 
saying that dom0 only frees up PV-guest-size minus existing-slack?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Mon, May 22, 2006 at  9:03 AM, in message
<6085df031157d9ce198b72191501db21@cl.cam.ac.uk>, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote: 
> On 22 May 2006, at 15:51, Charles Coffing wrote:
> 
>> HVM domains need some extra memory free for the shadow page tables,
>> otherwise they may crash the entire machine while they are running,
or
>> the HVM domain itself may crash (exact behavior depends on whether
you
>> have Yunhong''s patch to change BUGs to domain_crash).
>>
>> This slack space is calculated into the memory size for HVM
domains,
>> but what happens if you then start a PV domain afterwards?  Only
the
>> minimally required memory is freed up, then the PV domain takes it
all,
>> leaving the HVM domain with no slack == crash.
> 
> Where does the slack from the previous HVM guest startup go? Are you
> saying that dom0 only frees up PV- guest- size minus existing-
slack?

Correct.

Unless, of course, the HVM domain has eaten up some of the slack in the
mean-time, in which case dom0 frees up PV-guest-size minus
remaining-slack.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22 May 2006, at 16:15, Charles Coffing wrote:
>> Where does the slack from the previous HVM guest startup go? Are you
>
>> saying that dom0 only frees up PV- guest- size minus existing-
> slack?
>
> Correct.
>
> Unless, of course, the HVM domain has eaten up some of the slack in the
> mean-time, in which case dom0 frees up PV-guest-size minus
> remaining-slack.
The auto-ballooning logic is shagged then. xend should keep track of 
memory requirements (inc. max overheads) of every domain, then ensure 
dom0 memory usage is no greater than total memory minus sum of all 
other domains'' memory requirements. How hard can that be? (Too hard for
original author I guess ;-) ).

Anyway, I''m going to back out the "4MB slack" hack.
Auto-ballooning can
never be stable in its current form (consider a guest that temporarily 
balloons down from 128MB to 64MB, then another (HVM) guest starts, then 
it tries to balloon back to 128MB -- it''ll fail, and the HVM guest will
lose any headroom it required).

  -- Keir


_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel

On 22 May 2006, at 16:38, Keir Fraser wrote:
>> Unless, of course, the HVM domain has eaten up some of the slack in 
>> the
>> mean-time, in which case dom0 frees up PV-guest-size minus
>> remaining-slack.
>
> The auto-ballooning logic is shagged then. xend should keep track of 
> memory requirements (inc. max overheads) of every domain, then ensure 
> dom0 memory usage is no greater than total memory minus sum of all 
> other domains'' memory requirements. How hard can that be? (Too
hard
> for original author I guess ;-) ).
>
> Anyway, I''m going to back out the "4MB slack" hack.
Auto-ballooning
> can never be stable in its current form (consider a guest that 
> temporarily balloons down from 128MB to 64MB, then another (HVM) guest 
> starts, then it tries to balloon back to 128MB -- it''ll fail, and
the
> HVM guest will lose any headroom it required).
Actually, since there is an argument for keeping it for live migration, 
I''ll keep it for the time being. But it does sound like auto-ballooning
needs some more thought.

I''ll apply a suitable patch for ia64 shortly....

  -- Keir


_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel