libvirt users - Aug 2013 - Information needed regarding the libvirt version libvirt-1.0.6-1

Hi,

We have upgraded the libvirt version from libvirt-1.0.5 to libvirt-1.0.6-1
on our linux machine. After this upgrade "virsh cpu-stats
<domain>" command
is not working on our HardWare.

*virsh # cpu-stats test
error: Failed to retrieve CPU statistics for domain 'test'
error: internal error error accessing cgroup cpuacct for vcpu*
**
*# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  3       4       1
ns      0       1       1
cpu     1       6       1
cpuacct 7       3       1
memory  2       3       1
freezer 0       1       1
net_cls 0       1       1
blkio   0       1       1
*
*# chkconfig --list | grep cg
cgroups_mount   0:off   1:on    2:on    3:on    4:on    5:on    6:off*

We have also found that there is no "cgroup" directory created by
kenel under "/sys/fs/".  We suspect that the 'cgroup'
controllers are not
mounted properly and hence the issue is observed. While using the
libvirt-1.0.5 version below mentioned mount commands were sufficient to
mount cgroup controllers

*mkdir -p /dev/cgroup_cpuacct
mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct*

As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we
feel that the above commands are now not sufficient. We feel we have to do
some more steps get the 'cgroups' mounted properly on our hardware.

Since we are new to libvirt and can you please help us by providing some
pointers to fix the above issue and get cpu-stats working on our HardWare.
Since we have to fix this on top priority, can you please reply as soon as
possible. Please let us know if you need any further information from our
side.

Thanks in advance.

BR
Abhinay

On Wed, Aug 21, 2013 at 09:06:21AM +0200, arun abhinay
wrote:
> Hi,
> 
> We have upgraded the libvirt version from libvirt-1.0.5 to libvirt-1.0.6-1
> on our linux machine. After this upgrade "virsh cpu-stats
<domain>" command
> is not working on our HardWare.
> 
> *virsh # cpu-stats test
> error: Failed to retrieve CPU statistics for domain 'test'
> error: internal error error accessing cgroup cpuacct for vcpu*
> **
> *# cat /proc/cgroups
> #subsys_name    hierarchy       num_cgroups     enabled
> cpuset  3       4       1
> ns      0       1       1
> cpu     1       6       1
> cpuacct 7       3       1
> memory  2       3       1
> freezer 0       1       1
> net_cls 0       1       1
> blkio   0       1       1
> *
> *# chkconfig --list | grep cg
> cgroups_mount   0:off   1:on    2:on    3:on    4:on    5:on    6:off*
> 
> We have also found that there is no "cgroup" directory created by
> kenel under "/sys/fs/".  We suspect that the 'cgroup'
controllers are not
> mounted properly and hence the issue is observed. While using the
> libvirt-1.0.5 version below mentioned mount commands were sufficient to
> mount cgroup controllers
> 
> *mkdir -p /dev/cgroup_cpuacct
> mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct*
> 
> As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we
Actually it changed in version 1.0.5
> feel that the above commands are now not sufficient. We feel we have to do
> some more steps get the 'cgroups' mounted properly on our hardware.
> 
> Since we are new to libvirt and can you please help us by providing some
> pointers to fix the above issue and get cpu-stats working on our HardWare.
> Since we have to fix this on top priority, can you please reply as soon as
> possible. Please let us know if you need any further information from our
> side.
Libvirt will never attempt to mount any cgroups controllers. It delegates
this task to the base OS. Historically the 'cgconfig' init script would
be used. On distros which use systemd though, systemd will take care of
this task. There's a bit more info here

  http://libvirt.org/cgroups.html

Basically our recommendation is to mount all the controllers individually
at /sys/fs/cgroup/<controllername>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Hi Daniel,

Thanks a lot for your reply.

We have mounted all the cgoup controllers at 
"/dev/cgroup_<controller>" as
mentioned below.
ls /dev/ | grep cgroup
cgroup_cpu    cgroup_cpuacct       cgroup_cpuset         cgroup_mem

On further debugging we dound that the "VCPU" directories are not
being
created for our domains and hence the comamnd "cpu-stats
<domain>" is
failing.
Below is the snippet of logs collected with debug logs enabled for libvirt.

2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:708 : Make
group /machine/anusha.libvirt-qemu/vcpu0
2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:729 :
Makecontroller /dev/cgroup_cpu/machine/anusha.libvirt-qemu/vcpu0/
2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:745 : Failed
to create controller cpu for group

Where anusha --> is the domain which was created for libvirt.
 Can you please let us know if it is enough to mount the controllers
mentioned above or we have missed mounting of any other controller. Your
inputs would be of great help to us. Thanking you in advance.

BR
Abhinay
On Wed, Aug 21, 2013 at 1:39 PM, Daniel P. Berrange
<berrange@redhat.com>wrote:
> On Wed, Aug 21, 2013 at 09:06:21AM +0200, arun abhinay wrote:
> > Hi,
> >
> > We have upgraded the libvirt version from libvirt-1.0.5 to
> libvirt-1.0.6-1
> > on our linux machine. After this upgrade "virsh cpu-stats
<domain>"
> command
> > is not working on our HardWare.
> >
> > *virsh # cpu-stats test
> > error: Failed to retrieve CPU statistics for domain 'test'
> > error: internal error error accessing cgroup cpuacct for vcpu*
> > **
> > *# cat /proc/cgroups
> > #subsys_name    hierarchy       num_cgroups     enabled
> > cpuset  3       4       1
> > ns      0       1       1
> > cpu     1       6       1
> > cpuacct 7       3       1
> > memory  2       3       1
> > freezer 0       1       1
> > net_cls 0       1       1
> > blkio   0       1       1
> > *
> > *# chkconfig --list | grep cg
> > cgroups_mount   0:off   1:on    2:on    3:on    4:on    5:on    6:off*
> >
> > We have also found that there is no "cgroup" directory
created by
> > kenel under "/sys/fs/".  We suspect that the
'cgroup' controllers are not
> > mounted properly and hence the issue is observed. While using the
> > libvirt-1.0.5 version below mentioned mount commands were sufficient
to
> > mount cgroup controllers
> >
> > *mkdir -p /dev/cgroup_cpuacct
> > mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct*
> >
> > As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1,
we
>
> Actually it changed in version 1.0.5
>
> > feel that the above commands are now not sufficient. We feel we have
to
> do
> > some more steps get the 'cgroups' mounted properly on our
hardware.
> >
> > Since we are new to libvirt and can you please help us by providing
some
> > pointers to fix the above issue and get cpu-stats working on our
> HardWare.
> > Since we have to fix this on top priority, can you please reply as
soon
> as
> > possible. Please let us know if you need any further information from
our
> > side.
>
> Libvirt will never attempt to mount any cgroups controllers. It delegates
> this task to the base OS. Historically the 'cgconfig' init script
would
> be used. On distros which use systemd though, systemd will take care of
> this task. There's a bit more info here
>
>   http://libvirt.org/cgroups.html
>
> Basically our recommendation is to mount all the controllers individually
> at /sys/fs/cgroup/<controllername>
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-   
http://www.flickr.com/photos/dberrange/:|
> |: http://libvirt.org              -o-            
http://virt-manager.org:|
> |: http://autobuild.org       -o-        
http://search.cpan.org/~danberr/:|
> |: http://entangle-photo.org       -o-      
http://live.gnome.org/gtk-vnc:|
>