Xen users - Mar 2008 - Limits on memory which can be set using xm mem-set

Hello,

I have a machine with Xen 3.1.3 and 4 GB of memory, 192 MB used by dom0. After
starting a new domain with 3400 MB of memory, everything looks as expected.
However, if I mem-set the memory of domU to be 160 MB, xm info show more than
200 MB of "dark memory". That is, the difference between free memory
and
memory occupied by dom0+domU is greater than 200 MB. Now, when I connect to
the domU using xm console, I can see a weird message:

    Setting mem allocation to 412160 kiB

/proc/meminfo in domU is consistent with it and shows MemTotal: 412160 kB

However, in dom0, xm list still shows the domU is provided with 160 MB of
memory. So, Is there any limit on minimum amount of memory which can be
assigned to a domain using xm mem-set? Perhaps the limit is somehow related to
the memory assigned to a domain at the time of its creation... Is this
behaviour normal and deterministic or even described somewhere?

And one more, loosely related question: can mem-set be made synchronous so
that after xm mem-set ends ballooning is finished and memory occupied by the
domain can be assigned to another one? That is, something similar to -w option
of xm shutdown which blocks until the domain is down.

Many thanks for your answers or suggestions

Jirka

-- 
In the beginning the Universe was created. This has made a lot of people very
angry and been widely regarded as a bad move.
        -- Douglas Adams, "The Restaurant at the End of the Universe"

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I have a machine with Xen 3.1.3 and 4 GB of memory, 192 MB used by dom0.
> After starting a new domain with 3400 MB of memory, everything looks as
> expected.
OK.
> However, if I mem-set the memory of domU to be 160 MB, xm info 
> show more than 200 MB of "dark memory". That is, the difference
between
> free memory and memory occupied by dom0+domU is greater than 200 MB. Now,
> when I connect to the domU using xm console, I can see a weird message:
>
>     Setting mem allocation to 412160 kiB
>
> /proc/meminfo in domU is consistent with it and shows MemTotal: 412160 kB
Weird!
> However, in dom0, xm list still shows the domU is provided with 160 MB of
> memory.
Really weird...!

If you try "xm info", that''ll report how much free memory is
*really*
available for running domains.  This comes straight from Xen so it should be 
trustworthy.  Does it agree with what xm list is telling you?  I''m
betting
that it won''t and that it''ll confirm that the domain is
actually 400MB, just
as it claims.

Xen doesn''t overcommit memory, so there''s no way for a domain
to appear to
have more physical RAM than it''s using in the host system (unless the
guest
is buggy of course ;-).

What I suspect has happened is that somehow the ballooning didn''t do
what you
asked it to, but Xend just assumed that it succeeded and therefore updated 
the memory total it''s reporting for that domain.
> So, Is there any limit on minimum amount of memory which can be 
> assigned to a domain using xm mem-set? Perhaps the limit is somehow related
> to the memory assigned to a domain at the time of its creation... Is this
> behaviour normal and deterministic or even described somewhere?
I think possibly there used to be a very low limit (in the 16-32MB region) for 
a minimum ballooning size for a domain.  The idea was to stop a domain being 
shrunk so much that it crashed due to memory crunch.  Maybe there''s now
some
more sophisticated logic of this kind - that could be what''s causing
your
problem.

If, after doing the steps you described, you reissue xm mem-set commands to 
shrink the domain to 160MB again does anything happen?  What if it''s
< 160MB?
What if 160MB < xm mem-set < 400MB?  What if xm mem-set > 400? 

I''ve not heard of this behaviour in ballooning before, although to be
honest
I''d expect such an aggressive shrinking might upset the guest anyhow,
so it''s
not necessarily something you''d want to do in production!  Still, maybe
if I
have a look through the code later I''ll be able to spot a new check
that''s
tripping things up.  It really ought to warn you a bit more...

Anything interesting in /var/log/xen/xend.log or /var/log/xen/xen-debug.log?

Side note: you can do something like:

echo size_in_bytes > /proc/xen/balloon

within the guest to really make it resize and bypass all checks in Xend.  This 
still won''t register a change in size under xm list though, because it
won''t
tell Xend about it.  Not ideal :-(
> And one more, loosely related question: can mem-set be made synchronous so
> that after xm mem-set ends ballooning is finished and memory occupied by
> the domain can be assigned to another one? That is, something similar to -w
> option of xm shutdown which blocks until the domain is down.
I don''t think there''s a way to do this currently. 
It''s perfectly possible
technically, though.

Cheers,
Mark


-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Mark,

Thanks for your reply.
> I''ve not heard of this behaviour in ballooning before, although to
be honest
> I''d expect such an aggressive shrinking might upset the guest
anyhow, so it''s
> not necessarily something you''d want to do in production!
Well, of course, the guest gets upset if its memory is shrank too much. And
the lowest amount of memory is related to the maximum memory a domain was
started with. However, 160MB is quite fine for a guest originally started with
3.5GB but only if only few daemons are running in it. For example, for a
domain with 15GB, it seems the minimum is somewhere around 500MB.
> echo size_in_bytes > /proc/xen/balloon
This really helped much. I tried it and realized that I couldn''t shrink
the
memory below 400MB. So the balloon driver was the culprit. I looked into it
and found minimum_target() function, which counts minimum memory based on
domain''s maximum memory. However, it''s worth noting that the
kernel is 2.6.21
by SuSE and there is no such function in 2.6.18 kernel from Xen 3.2.0
distribution. So the problem is only related to SuSE kernels.

Jirka

-- 
Theorem: a cat has nine tails.
Proof:
    No cat has eight tails. A cat has one tail more than no cat.
    Therefore, a cat has nine tails.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Thanks for your reply.
>
> > I''ve not heard of this behaviour in ballooning before,
although to be
> > honest I''d expect such an aggressive shrinking might upset
the guest
> > anyhow, so it''s not necessarily something you''d want
to do in production!
>
> Well, of course, the guest gets upset if its memory is shrank too much. And
> the lowest amount of memory is related to the maximum memory a domain was
> started with. However, 160MB is quite fine for a guest originally started
> with 3.5GB but only if only few daemons are running in it. For example, for
> a domain with 15GB, it seems the minimum is somewhere around 500MB.
Ok, well if it works for you then fair enough.

I''ve seen Linux''s memory management system get upset when it
actually has
plenty of memory to spare, seemingly because it gets scared when the 
available memory decreases by a large amount :-)  Note that the problem I saw 
wasn''t not having /enough/ memory in the guest, it was that the guest
went
into emergency behaviours simply because the memory had suddenly shrunk...

Memory ballooning gets seen by the guest as a big memory allocation, so this 
behaviour is kind of understandable.  The balloon driver is less aggressive 
these days so Linux is probably less scared by it.  Still, I''d advise
to
verify that you can reliably do aggressive ballooning stably before using it 
in production.

This may be less of a problem these days, of course - the balloon drivers I 
managed to really provoke into breaking things were very old indeed.

Note also that a domain that''s created with 4G and resized to 400M will
have /less available memory/ than one that is created for 400M in the first 
place.  That''s because the OS will have created larger data structures
to
cope with the larger memory (as pointed out in one of the threads below) - 
again something to bear in mind.
> > echo size_in_bytes > /proc/xen/balloon
>
> This really helped much.
Ah, glad to hear it :-)
> I tried it and realized that I couldn''t shrink the 
> memory below 400MB. So the balloon driver was the culprit. I looked into it
> and found minimum_target() function, which counts minimum memory based on
> domain''s maximum memory. However, it''s worth noting that
the kernel is
> 2.6.21 by SuSE and there is no such function in 2.6.18 kernel from Xen
> 3.2.0 distribution. So the problem is only related to SuSE kernels.
Coincidentially this got recently mentioned on xen-devel.  SuSE provide a 
patch in their balloon driver to prevent over aggressive ballooning, so your 
diagnosis is indeed correct.

Here are some relevant threads:
http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00143.html
http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00116.html

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users