Pkg xen devel - Sep 2011 - Bug#640500: xen-hypervisor-4.0-amd64: xend invokes oomkiller and reboots machine when creating DomU's

Package: xen-hypervisor-4.0-amd64
Version: 4.0.1-2
Severity: normal
Tags: upstream

When creating Xen DomU's, at some point xend invokes the oom-killer and
the entire machine restarts:

Sep  5 12:04:59 arbiter kernel: [259697.101212] __ratelimit: 136
callbacks suppressed
Sep  5 12:04:59 arbiter kernel: [259697.101218] xend invoked oom-killer:
gfp_mask=0x200da, order=0, oom_adj=0
Sep  5 12:04:59 arbiter kernel: [259697.101225] xend cpuset=/
mems_allowed=0
Sep  5 12:04:59 arbiter kernel: [259697.101230] Pid: 1841, comm: xend
Not tainted 2.6.32-5-xen-amd64 #1
Sep  5 12:06:23 arbiter kernel: imklog 4.6.4, log source = /proc/kmsg
started.
Sep  5 12:06:23 arbiter rsyslogd: [origin software="rsyslogd"
swVersion="4.6.4" x-pid="1409"
x-info="http://www.rsyslog.com"]
(re)start
Sep  5 12:06:23 arbiter kernel: [    0.000000] Initializing cgroup
subsys cpuset

xm info says the machine has 10231 MB of RAM. These are the VMs that
normally run:

Name                     ID   Mem VCPUs      State Time(s)
Domain-0                 0  2177     1     r----- 127.1
one                      1  1536     2     r----- 192.4
two                      2   512     1     -b---- 2.7
three                    3  1024     1     -b---- 188.6
four                     4   512     1     -b---- 12.7
five                     5  2048     2     -b---- 151.7
six                      6  1024     2     r----- 64.9
seven                    7  1024     2     -b---- 65.9
eight                    8  1024     2     -b---- 100.7

That means 8704 MB of memory is used. 

Over the last couple of days, I've experienced that when creating two
VM's with 512 MB RAM, the oom-killer is invoked. This should not happen
because Dom0 can shrink and because the machine should not restart when
it is out of memory.

This is a production Xen host, which is becomming a liability, because
it's so unpredictable.

Current Dom0 kernel: Linux arbiter 2.6.32-5-xen-amd64 

-- System Information:
Debian Release: 6.0.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

xen-hypervisor-4.0-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.0-amd64 recommends:
ii  xen-utils-4.0                 4.0.1-2    XEN administrative tools

Versions of packages xen-hypervisor-4.0-amd64 suggests:
pn  xen-docs-4.0                  <none>     (no description available)

-- no debconf information

Severity is set to normal, but I set it to grave or serious in the bug 
report tool. This bug is more severe than 'normal', in any case.

On 09/05/2011 06:48 PM, Wiebe Cazemier wrote:
> Package: xen-hypervisor-4.0-amd64
> Version: 4.0.1-2
> Severity: normal
> Tags: upstream
> 
> When creating Xen DomU's, at some point xend invokes the oom-killer and
> the entire machine restarts:
> 
> Sep  5 12:04:59 arbiter kernel: [259697.101212] __ratelimit: 136
> callbacks suppressed
> Sep  5 12:04:59 arbiter kernel: [259697.101218] xend invoked oom-killer:
> gfp_mask=0x200da, order=0, oom_adj=0
> Sep  5 12:04:59 arbiter kernel: [259697.101225] xend cpuset=/
> mems_allowed=0
> Sep  5 12:04:59 arbiter kernel: [259697.101230] Pid: 1841, comm: xend
> Not tainted 2.6.32-5-xen-amd64 #1
> Sep  5 12:06:23 arbiter kernel: imklog 4.6.4, log source = /proc/kmsg
> started.
> Sep  5 12:06:23 arbiter rsyslogd: [origin software="rsyslogd"
> swVersion="4.6.4" x-pid="1409"
x-info="http://www.rsyslog.com"]
> (re)start
> Sep  5 12:06:23 arbiter kernel: [    0.000000] Initializing cgroup
> subsys cpuset
> 
> xm info says the machine has 10231 MB of RAM. These are the VMs that
> normally run:
> 
> Name                     ID   Mem VCPUs      State Time(s)
> Domain-0                 0  2177     1     r----- 127.1
> one                      1  1536     2     r----- 192.4
> two                      2   512     1     -b---- 2.7
> three                    3  1024     1     -b---- 188.6
> four                     4   512     1     -b---- 12.7
> five                     5  2048     2     -b---- 151.7
> six                      6  1024     2     r----- 64.9
> seven                    7  1024     2     -b---- 65.9
> eight                    8  1024     2     -b---- 100.7
Funny domU names! :)
> That means 8704 MB of memory is used. 
> 
> Over the last couple of days, I've experienced that when creating two
> VM's with 512 MB RAM, the oom-killer is invoked. This should not happen
> because Dom0 can shrink and because the machine should not restart when
> it is out of memory.
> 
> This is a production Xen host, which is becomming a liability, because
> it's so unpredictable.
> 
> Current Dom0 kernel: Linux arbiter 2.6.32-5-xen-amd64 
Hi,

I wouldn't recommend to just let your dom0 shrink. Set it directly to a
much lower memory footprint, something like 512 MB of RAM, so that the
Linux kernel sees directly that it doesn't have so much memory to deal
with, and it will not allocate too big buffers and so on. Here's how to
do this with the Squeeze grub:

                echo "
# Start dom0 with less RAM
GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=512M\"
" >>/etc/default/grub

Then either run update-grub2 or dpkg-reconfigure grub-pc (and of course
reboot...). If you do that, I'm sure your troubles will go away. You
shouldn't be running big software in the dom0 anyway. I always done that
because of habits from running older Xen (since 2.0.7), and because I've
seen issues with the dom0 ballooning out. To me, ballooning the memory
of dom0 is more a desktop feature than for a server.

I believe this bug should be closed, because that's not really an issue
with Xen (but more with the way the Linux dom0 kernel works). I'll let
Bastian decide though.

Thomas Goirand (zigo)

On Tue, Sep 06, 2011 at 12:54:32AM +0800, Thomas Goirand
wrote:
> 
> > That means 8704 MB of memory is used. 
> > 
> > Over the last couple of days, I''ve experienced that when
creating two
> > VM''s with 512 MB RAM, the oom-killer is invoked. This should
not happen
> > because Dom0 can shrink and because the machine should not restart
when
> > it is out of memory.
> > 
> > This is a production Xen host, which is becomming a liability, because
> > it''s so unpredictable.
> > 
> > Current Dom0 kernel: Linux arbiter 2.6.32-5-xen-amd64 
> 
> Hi,
> 
> I wouldn''t recommend to just let your dom0 shrink. Set it directly
to a
> much lower memory footprint, something like 512 MB of RAM, so that the
> Linux kernel sees directly that it doesn''t have so much memory to
deal
> with, and it will not allocate too big buffers and so on. Here''s
how to
> do this with the Squeeze grub:
> 
>                 echo "
> # Start dom0 with less RAM
> GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=512M\"
> " >>/etc/default/grub
> 
> Then either run update-grub2 or dpkg-reconfigure grub-pc (and of course
> reboot...). If you do that, I''m sure your troubles will go away.
You
> shouldn''t be running big software in the dom0 anyway. I always
done that
> because of habits from running older Xen (since 2.0.7), and because
I''ve
> seen issues with the dom0 ballooning out. To me, ballooning the memory
> of dom0 is more a desktop feature than for a server.
> 
> I believe this bug should be closed, because that''s not really an
issue
> with Xen (but more with the way the Linux dom0 kernel works). I''ll
let
> Bastian decide though.
> 
> Thomas Goirand (zigo)
> 
Related link / dom0_mem:
http://wiki.xen.org/xenwiki/XenBestPractices

-- Pasi

On 09/08/2011 01:44 AM, Pasi K?rkk?inen wrote:
> On Tue, Sep 06, 2011 at 12:54:32AM +0800, Thomas Goirand wrote:
>>                 echo "
>> # Start dom0 with less RAM
>> GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=512M\"
>> " >>/etc/default/grub
>>
>> Then either run update-grub2 or dpkg-reconfigure grub-pc (and of course
>> reboot...). If you do that, I''m sure your troubles will go
away. You
>> shouldn''t be running big software in the dom0 anyway. I always
done that
>> because of habits from running older Xen (since 2.0.7), and because
I''ve
>> seen issues with the dom0 ballooning out. To me, ballooning the memory
>> of dom0 is more a desktop feature than for a server.
>>
>> I believe this bug should be closed, because that''s not really
an issue
>> with Xen (but more with the way the Linux dom0 kernel works).
I''ll let
>> Bastian decide though.
>>
>> Thomas Goirand (zigo)
>>
> 
> Related link / dom0_mem:
> http://wiki.xen.org/xenwiki/XenBestPractices
> 
> -- Pasi
Hi,

This is an "Immutable Page", which I guess means read only. Do you
know
how we could add the info about grub2 in Debian, with what I wrote above?

Cheers,

Thomas

Your message dated Wed, 21 Dec 2011 01:07:16 +0800
with message-id <4EF0C0C4.9040008 at goirand.fr>
and subject line There's no issue, just user not using Xen correctly, so
closing.
has caused the Debian Bug report #640500,
regarding xen-hypervisor-4.0-amd64: xend invokes oomkiller and reboots machine
when creating DomU's
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
640500: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640500
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Wiebe Cazemier <wiebe at ytec.nl>
Subject: xen-hypervisor-4.0-amd64: xend invokes oomkiller and reboots machine
when creating DomU's
Date: Mon, 05 Sep 2011 12:48:10 +0200
Size: 4071
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20111220/779ca942/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Thomas Goirand <thomas at goirand.fr>
Subject: There's no issue, just user not using Xen correctly, so closing.
Date: Wed, 21 Dec 2011 01:07:16 +0800
Size: 2538
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20111220/779ca942/attachment-0001.mht>