libvirt users - Sep 2014 - Re: cgroups inside LXC containers losts memory limits after some time

Hi all

I have CentOS Linux release 7.0.1406, libvirt 1.2.7 installed.
Just after create and start inside LXC container present cgroups.
Example for memory:

[root@ce7-t1 /]# ls -la /sys/fs/cgroup/memory/
total 0
drwxr-xr-x  2 root root   0 Sep 15 17:14 .
drwxr-xr-x 12 root root 280 Sep 15 17:14 ..
-rw-r--r--  1 root root   0 Sep 15 17:14 cgroup.clone_children
--w--w--w-  1 root root   0 Sep 15 17:14 cgroup.event_control
-rw-r--r--  1 root root   0 Sep 15 17:15 cgroup.procs
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.failcnt
--w-------  1 root root   0 Sep 15 17:14 memory.force_empty
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.failcnt
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.limit_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.max_usage_in_bytes
-r--r--r--  1 root root   0 Sep 15 17:14 memory.kmem.slabinfo
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.tcp.failcnt
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.tcp.limit_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.kmem.tcp.max_usage_in_bytes
-r--r--r--  1 root root   0 Sep 15 17:14 memory.kmem.tcp.usage_in_bytes
-r--r--r--  1 root root   0 Sep 15 17:14 memory.kmem.usage_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.limit_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.max_usage_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.memsw.failcnt
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.memsw.limit_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.memsw.max_usage_in_bytes
-r--r--r--  1 root root   0 Sep 15 17:14 memory.memsw.usage_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.move_charge_at_immigrate
-r--r--r--  1 root root   0 Sep 15 17:14 memory.numa_stat
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.oom_control
----------  1 root root   0 Sep 15 17:14 memory.pressure_level
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.soft_limit_in_bytes
-r--r--r--  1 root root   0 Sep 15 17:14 memory.stat
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.swappiness
-r--r--r--  1 root root   0 Sep 15 17:14 memory.usage_in_bytes
-rw-r--r--  1 root root   0 Sep 15 17:14 memory.use_hierarchy
-rw-r--r--  1 root root   0 Sep 15 17:14 notify_on_release
-rw-r--r--  1 root root   0 Sep 15 17:14 tasks

Command "free" inside LXC showed almost normal values:
[root@ce7-t1 /]# free
             total       used       free     shared    buffers     cached
Mem:       1048576      32972    1015604    4473848          0   -4445364
-/+ buffers/cache:    4478336   -3429760
Swap:      1048576          0    1048576

(some problem with negative values)

After unpredictable time passed (1-5 day ?), cgroups inside LXC
magicaly removed. "free" in such containers show 2^53-1 as maximum
values:
[root@puppet01 /]# free
             total       used       free     shared    buffers     cached
Mem:    9007199254740991     591180 9007199254149811          0
  0     267924
-/+ buffers/cache:     323256 9007199254417735
Swap:            0          0          0

And no more any cgroups presented at least in memory category:
[root@puppet01 /]# ls -la   /sys/fs/cgroup/memory/
total 0


b.r.
 Maxim Kozin

HI all
>After unpredictable time passed (1-5 day ?), cgroups inside LXC
>magicaly removed.
virsh dumpxml config look like this:
<domain type='lxc' id='3566'>
  <name>puppet</name>
  <uuid>6d49b280-5686-4e3c-b048-1b5d362fb137</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>8388608</hard_limit>
    <soft_limit unit='KiB'>8388608</soft_limit>
    <swap_hard_limit unit='KiB'>9437184</swap_hard_limit>
  </memtune>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='block' accessmode='passthrough'>
      <source dev='/dev/data/puppet'/>
      <target dir='/'/>
    </filesystem>
    <interface type='direct'>
      <mac address='02:00:00:25:a7:56'/>
      <source dev='br502' mode='bridge'/>
      <model type='virtio'/>
    </interface>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='lxc' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
  <seclabel type='none'/>
</domain>

Problem go away after restart LXC container. But occurred again after
some time passed.
  For containers with broken cgroups virsh memtune fails with error:
[root@]# virsh memtune --hard-limit 8G puppet    --live --config
error: Unable to change memory parameters
error: unable to set memory hard_limit tunable: Operation not permitted

b.r.
 Maxim Kozin

>>After unpredictable time passed (1-5 day ?), cgroups inside LXC
>>magicaly removed.
I use next workaround for such containers:

1) restore subgroup in cgroup hierarchy :
[root@]# mkdir /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope

2) put value in limits:
[root@]# echo 8589934592 >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.limit_in_bytes

[root@]# echo 9663676416  >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dphosgene7.infra.scope/memory.memsw.limit_in_bytes

[root@]# echo 8589934592  >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.kmem.limit_in_bytes

[root@]# echo 8589934592  >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.kmem.tcp.limit_in_bytes

[root@]# echo 8589934592  >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.soft_limit_in_bytes

3) check that inside container  limits still unset:
[root@puppet01 /]# free
             total       used       free     shared    buffers     cached
Mem:    9007199254740991    1591360 9007199253149631          0
  0     913112
-/+ buffers/cache:     678248 9007199254062743
Swap:            0          0          0

4) get libvirtd pid of such container:
[root@]# ps ax | grep puppet | grep libvirt
 3142 ?        Ssl    0:17 /usr/libexec/libvirt_lxc --name puppet
--console 21 --security=none --handshake 27 --background --veth
macvlan0
Moreover, you need get pid of all process existed in LXC.  We would test
in ssh sessions, because get pid of sshd run in our container.
[root@]# pstree -ap 3142 | grep ssh
  |   `-sshd,5527


5) Assign pid with cgroups:
[root@]# cgclassify -g
memory:machine.slice/machine-lxc\\x2dpuppet.scope  3142 3143 5527

(I put pid 3143 in list - it's init inside LXC)

6) check again in container - start new ssh session:
[root@puppet01 /]# free
             total       used       free     shared    buffers     cached
Mem:       8388608        392    8388216          0          0          0
-/+ buffers/cache:        392    8388216
Swap:      1048576          0    1048576


7) compile and run loadmem:

#include <stdlib.h>

int main() {
  int *p;
  while(1) {
    int inc=1024*1024*sizeof(char);
    p=(int*) calloc(1,inc);
    if(!p) break;
  }
}

After 8G allocated process loadmem would be killed

b.r.
 Maxim Kozin