libvirt users - Jun 2013 - Re: cgroup error starting domains

On Fri, Jun 07, 2013 at 01:07:01PM +0200, Matteo Bernardini
wrote:
> Thanks Daniel for helping with this :)
> 
> # cat /proc/mounts
[snip]
> cgroup /sys/fs/cgroup cgroup 
rw,relatime,bfqio,hugetlb,perf_event,blkio,net_cls,freezer,devices,memory,cpuacct,cpu,cpuset,clone_children
Ok so here you've mounted all the cgroups controllers at the same
place. This is really strongly recommended against. It is better
to mount each controller separately at

   /sys/fs/cgroup/<controller name>

and have /sys/fs/cgroup as a tmpfs.

http://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups

eg to get something along the lines of:

$ ls -l /sys/fs/cgroup/
total 0
drwxr-xr-x. 3 root root  0 Jun  3 09:41 blkio
lrwxrwxrwx. 1 root root 11 Jun  3 09:41 cpu -> cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Jun  3 09:41 cpuacct -> cpu,cpuacct
drwxr-xr-x. 3 root root  0 Jun  3 09:41 cpu,cpuacct
drwxr-xr-x. 3 root root  0 Jun  3 09:41 cpuset
drwxr-xr-x. 3 root root  0 Jun  3 09:41 devices
drwxr-xr-x. 3 root root  0 Jun  3 09:41 freezer
drwxr-xr-x. 3 root root  0 Jun  3 09:41 memory
drwxr-xr-x. 2 root root  0 Jun  3 09:41 net_cls
drwxr-xr-x. 2 root root  0 Jun  3 09:41 perf_event
drwxr-xr-x. 4 root root  0 Jun  3 09:42 systemd
$ grep cgroup /proc/mounts 
tmpfs /sys/fs/cgroup tmpfs rw,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0


(obviously the 'system' mount point is not required if not using
systemd).
> in /etc/libvirt/qemu.conf I uncommented this line
> 
> cgroup_controllers = [ "cpu", "devices",
"memory", "blkio", "cpuset",
> "cpuacct" ]
And this, together with your mount setup is what causes the problem.

Here you have told libvirt to *only* use those 6 listed cgroups controllers,
 but because you've mounted all 9 controllers at the same mount point,
it is impossible for libvirt to satisfy  the config you requested. It can't
use the 6 controllers you asked for, without also using the other 3 you have
excluded in the config.

Either comment out that cgroup_controllers setting (to let libvirt do
auto-detection of mounts), or switch to using a separate mount per
controller, or both.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Tried with commenting out cgroup_controllers but with no change.

Tried also unmounting /sys/fs/cgroup and mount the cgroups in a subfolder
hierarchy manually but it remounted still all together (I think this is
related at the unmounting support going away, I'm with kernel 3.10-rc4 on
the test machine).

So I had a look at Slackware's init system's scripts: /sys/fs/cgroup is
mounted in /etc/rc.d/rc.S, the main one, so I tried changed it to automount
in subfolders the available controllers

--- ./etc/rc.d/rc.S.orig        2012-09-13 21:38:34.000000000 +0200
+++ ./etc/rc.d/rc.S     2013-06-07 19:56:06.176366278 +0200
@@ -51,7 +51,14 @@
 # Mount Control Groups filesystem interface:
 if grep -wq cgroup /proc/filesystems ; then
   if [ -d /sys/fs/cgroup ]; then
-    mount -t cgroup cgroup /sys/fs/cgroup
+    # Mount a tmpfs as the cgroup filesystem root
+    /sbin/mount -t tmpfs cgroup_root /sys/fs/cgroup
+    # Mount individual cgroup controllers:
+    controllers="$(/usr/bin/lssubsys -a 2>/dev/null | tr '\n'
' ' |
/usr/bin/sed s/.$//)"
+    for i in $controllers; do
+      /usr/bin/mkdir /sys/fs/cgroup/$i
+      /sbin/mount -t cgroup -o $i $i /sys/fs/cgroup/$i
+    done
   else
     mkdir -p /dev/cgroup
     mount -t cgroup cgroup /dev/cgroup

This way virtual machines start fine, thanks a lot Daniel :)

Matteo

just FYI, I just read this

http://lists.freedesktop.org/archives/systemd-devel/2013-June/011521.html

I just hope I can still use libvirt and lxc without systemd: if not, also
if I'm using them for my services since many years, I'll be forced to
drop
both.

Matteo