libvirt users - Dec 2012 - RHEL6 cgroup error after a few days of uptime

I have a RHEL6 that hosts many kvm virtual machines. It has been
working fine for a couple years. I apply errata updates about once a
week.

In the last couple weeks, I've ran into a bug where the virtual
machines start failing to start with a cgroup error message. If I
reboot the host (very disruptive) then things start working normaly
for a few days.

Can I configure qemu/libvirt not to use cgroups at all as a temporary
workaround? How would I do that?

Here is example of trying to start a vm (that hasn't been started
since boot) after the problem manifests itself.

[root at virthost ~]# find /cgroup | grep r04s14
[root at virthost ~]# virsh start r04s14
error: Failed to start domain r04s14
error: Unable to create cgroup for r04s14: No such file or directory
[root at virthost ~]# find /cgroup | grep r04s14
/cgroup/cpu/libvirt/qemu/r04s14
/cgroup/cpu/libvirt/qemu/r04s14/cpu.rt_period_us
/cgroup/cpu/libvirt/qemu/r04s14/cpu.rt_runtime_us
/cgroup/cpu/libvirt/qemu/r04s14/cpu.stat
/cgroup/cpu/libvirt/qemu/r04s14/cpu.cfs_period_us
/cgroup/cpu/libvirt/qemu/r04s14/cpu.cfs_quota_us
/cgroup/cpu/libvirt/qemu/r04s14/cpu.shares
/cgroup/cpu/libvirt/qemu/r04s14/cgroup.event_control
/cgroup/cpu/libvirt/qemu/r04s14/notify_on_release
/cgroup/cpu/libvirt/qemu/r04s14/cgroup.procs
/cgroup/cpu/libvirt/qemu/r04s14/tasks
/cgroup/cpuacct/libvirt/qemu/r04s14
/cgroup/cpuacct/libvirt/qemu/r04s14/cpuacct.stat
/cgroup/cpuacct/libvirt/qemu/r04s14/cpuacct.usage_percpu
/cgroup/cpuacct/libvirt/qemu/r04s14/cpuacct.usage
/cgroup/cpuacct/libvirt/qemu/r04s14/cgroup.event_control
/cgroup/cpuacct/libvirt/qemu/r04s14/notify_on_release
/cgroup/cpuacct/libvirt/qemu/r04s14/cgroup.procs
/cgroup/cpuacct/libvirt/qemu/r04s14/tasks
[root at virthost ~]#

On Thu, Dec 13, 2012 at 01:31:27PM -0700, Dax Kelson
wrote:
> I have a RHEL6 that hosts many kvm virtual machines. It has been
> working fine for a couple years. I apply errata updates about once a
> week.
> 
> In the last couple weeks, I've ran into a bug where the virtual
> machines start failing to start with a cgroup error message. If I
> reboot the host (very disruptive) then things start working normaly
> for a few days.
Sounds like something has either unmounted your cgroups, or
deleted the directories that libvirt created. I wonder if the
cgconfig initscript is doing it perhaps, and getting triggered in
a %post from an RPM script. In any case, you ought not need to
reboot the host - restarting libvirtd will get it to re-create
its cgroups.
> Can I configure qemu/libvirt not to use cgroups at all as a temporary
> workaround? How would I do that?
Simply don't mount any on the host and libvirt won't use them.
On RHEL6, the 'cgconfig' initscript is what mounts them at boot.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|