libvirt users - Aug 2013 - Re: virsh "cpu-stats" not working !!!

>
> Hi All,
>
> Can you please guide us in solving this issue.
>
> Version used : libvirt-1.0.6
> # rpm -qa | egrep  "libvirt|qemu"
> qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64
> libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
>
> Problem :
>
> # virsh cpu-stats CLA-0
> error: Failed to retrieve CPU statistics for domain 'CLA-0'
> *error: internal error error accessing cgroup cpuacct for vcpu*
>
> Related info :
>
> # cat /proc/cgroups
> #subsys_name    hierarchy       num_cgroups     enabled
> cpuset                      3                     4                  1
> ns                             0                    1                  1
> cpu                           1                    6                  1
> cpuacct                     7                    3                  1
> memory                     2                    3                  1
> freezer                       0                    1                  1
> net_cls                      0                    1                  1
> blkio                          0                    1                  1
>
> # cat /proc/mounts | grep cpu
> cpu /dev/cgroup_cpu cgroup rw,relatime,cpu 0 0
> cpuset /dev/cgroup_cpuset cgroup rw,relatime,cpuset 0 0
> cpuacct /dev/cgroup_cpuacct cgroup rw,relatime,cpuacct 0 0
>
> # cat /proc/self/cgroup
> 7:cpuacct:/
> 3:cpuset:/
> 2:memory:/
> 1:cpu:/
>
> virsh # vcpucount CLA-0
> maximum      config         3
> maximum      live           3
> current      config         3
> current      live           3
>
> virsh # vcpuinfo CLA-0
> VCPU:           0
> CPU:            0
> State:          running
> CPU time:       5.7s
> CPU Affinity:   yyyyyyyy
>
> virsh # vcpupin CLA-0 --current
> VCPU: CPU Affinity
> ----------------------------------
>    0: 0-7
>    1: 0-7
>    2: 0-7
>
> # cat /dev/cgroup_cpuacct/cpuacct.usage_percpu
> 532996776861 117928641102 125174572624 167825543709 155234053284
> 97350309867 122372010702 110640812251
> # cat /dev/cgroup_cpuacct/cpuacct.usage
> 1431288376064
> # cat /dev/cgroup_cpuacct/cpuacct.stat
> user 12148
> system 81469
>
> # cat /dev/cgroup_cpuacct/*machine*/cpuacct.usage_percpu
> 0 0 0 0 0 0 0 0
> # cat /dev/cgroup_cpuacct/*machine/*cpuacct.usage
> 0
>
> # chkconfig --list | grep cg
> cgroups_mount   0:off   1:on    2:on    3:on    4:on    5:on    6:off
>
> We have also found that there is no "*cgroup*" directory created
under *
> "/sys/fs/".* We have tried mounting the same manually using the
below
> commands but we were not successful.
>
> mkdir -p /dev/cgroup_cpuacct
> mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct
>
> (or)
>
>  mkdir -p /sys/fs/cgroup
> mount -t cgroup -o cgroup cgroup /sys/fs/cgroup
>
>
> Please find the the domain info used for the creation of VM in the
> attachment. Please let us know, how to proceed futher.
>
> Your inputs would be of great help to us.
>
> Thanks and Regards,
> Shree Duth Awasthi.
>

On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote:

[putting your name in ALL CAPS looks like you are shouting]
>>
>> Hi All,
>>
>> Can you please guide us in solving this issue.
>>
>> Version used : libvirt-1.0.6
>> # rpm -qa | egrep  "libvirt|qemu"
>> qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64
>> libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
>>
>> Problem :
>>
>> # virsh cpu-stats CLA-0
>> error: Failed to retrieve CPU statistics for domain 'CLA-0'
>> *error: internal error error accessing cgroup cpuacct for vcpu*
Are you sure cpuacct was mounted prior to libvirtd starting?
>>
>> Related info :
>>
>> # cat /proc/cgroups
>> #subsys_name    hierarchy       num_cgroups     enabled
>> cpuset                      3                     4                  1
>> ns                             0                    1                 
1
>> cpu                           1                    6                  1
>> cpuacct                     7                    3                  1
The fact that it has a rather high hierarchy number (7) makes me think
you mounted it after the fact.

>>
>> We have also found that there is no "*cgroup*" directory
created under *
>> "/sys/fs/".* We have tried mounting the same manually using
the below
>> commands but we were not successful.
Cgroups are typically mounted by systemd in newer systems, but the fact
that you are trying to mount manually says that you aren't using
systemd.  Libvirt won't mount cgroups; it only uses what was mounted at
the time it was started.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Hi Eric,

Thanks for your valuable time and comments.
>> Are you sure cpuacct was mounted prior to libvirtd starting?
Yes, we are mounting cpuacct before starting the libvirtd daemon.
>> Libvirt won't mount cgroups;
Yes, Agree.
>>it only uses what was mounted at the time it was started.
But we are mounting it well before it is started.

We are facing the issue only while executing the "cpu-stats <domain
name>"
command.

One more observation is that : While starting the domain, we are observing
the below error.

"2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953
: Unable to get vcpus' pids."

 This may be the root cause of  virsh cpu-stats not working. ( seems to be
because of CPU pinning issue )

Because of this the VCPU directories are not created for the corresponding
domain in
"/dev/cgroup_<controller>/machine/<domain>.libvirt-qemu/"
and
hence the "cpu-stats" command is not working.
Please let us know your view on this.

We  have collected the debug logs, please find the attachment for the same..

Thanks and Regards,
Shree Duth Awasthi.

On Wed, Aug 21, 2013 at 9:35 PM, Eric Blake <eblake@redhat.com> wrote:
> On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote:
>
> [putting your name in ALL CAPS looks like you are shouting]
>
> >>
> >> Hi All,
> >>
> >> Can you please guide us in solving this issue.
> >>
> >> Version used : libvirt-1.0.6
> >> # rpm -qa | egrep  "libvirt|qemu"
> >> qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64
> >> libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
> >>
> >> Problem :
> >>
> >> # virsh cpu-stats CLA-0
> >> error: Failed to retrieve CPU statistics for domain
'CLA-0'
> >> *error: internal error error accessing cgroup cpuacct for vcpu*
>
> Are you sure cpuacct was mounted prior to libvirtd starting?
>
> >>
> >> Related info :
> >>
> >> # cat /proc/cgroups
> >> #subsys_name    hierarchy       num_cgroups     enabled
> >> cpuset                      3                     4               
1
> >> ns                             0                    1             
1
> >> cpu                           1                    6              
1
> >> cpuacct                     7                    3                
1
>
> The fact that it has a rather high hierarchy number (7) makes me think
> you mounted it after the fact.
>
>
> >>
> >> We have also found that there is no "*cgroup*" directory
created under *
> >> "/sys/fs/".* We have tried mounting the same manually
using the below
> >> commands but we were not successful.
>
> Cgroups are typically mounted by systemd in newer systems, but the fact
> that you are trying to mount manually says that you aren't using
> systemd.  Libvirt won't mount cgroups; it only uses what was mounted at
> the time it was started.
>
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>