libvirt users - Mar 2014 - cgroup for VM - does it work properly?

Hi

I tried to restrict usage of some running VM by cpu.shares (i.e. set to
10 from original 1024) on loaded system and it seem doesn't work as I
expected... all running processes has same CPU usage (by htop) :-/
Does anyone has same experience?

Fedora 19, libvirt-1.0.5.9-1.fc19.x86_64

Thanks a lot
Martin

-- 
Martin Pavlásek <mpavlase@redhat.com>
 OpenStack QA Associate/Red Hat Czech/BRQ
 irc: mpavlase

On 03/05/2014 04:32 PM, Martin Pavlásek wrote:
> Hi
> 
> I tried to restrict usage of some running VM by cpu.shares (i.e. set to
> 10 from original 1024) on loaded system and it seem doesn't work as I
> expected... all running processes has same CPU usage (by htop) :-/
How exactly are you changing the cgroup settings?  Are you trying to
directly modify cgroup files (unsupported), or using the libvirt API
(such as via virsh schedinfo)?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

On Thu 06 Mar 2014 00:42:27 CET, Eric Blake wrote:
> On 03/05/2014 04:32 PM, Martin Pavlásek wrote:
>> Hi
>>
>> I tried to restrict usage of some running VM by cpu.shares (i.e. set to
>> 10 from original 1024) on loaded system and it seem doesn't work as
I
>> expected... all running processes has same CPU usage (by htop) :-/
>
> How exactly are you changing the cgroup settings?  Are you trying to
> directly modify cgroup files (unsupported), or using the libvirt API
> (such as via virsh schedinfo)?
>
Via mounted virtualfs - 
/sys/fs/cgroup/cpu/machine/<name-of-VM>/{emulator,vcpu0}/cpu.shares

--
--
Martin Pavlásek <mpavlase@redhat.com>
 OpenStack QA Associate/Red Hat Czech/BRQ
 irc: mpavlase

Hi Eric,

Sorry to hijack this thread.

Have you used vcpu_quota in production? I have come across a CPU stall
problem when vcpu_quota is effective not long ago and I'm still looking for
an answer.

For example, when I set
virsh schedinfo <id> --set vcpu_quota=5000

Then no matter the vm is loaded or idle, in about two weeks, sometimes
sooner, the vm will have CPU stall in dmesg log. The CPU stall problem
makes system not responsible and results in network failure, i.e. ping and
ssh will fail. I have tried a few guest distro (ubuntu, CentOS, debian)
with different kernel version (3.2, 3.8) and they all have this problem.

Do you have any idea what's behind the scenes?

The software I am running:
hypervisor os: ubuntu 12.04.3
hypervisor kernel: 3.8.0-35-generic
vm os: ubuntu 12.04  // I have tried other OSes.
vm kernel: 3.2.0-56-virtual
libvirt: 1.1.1-0ubuntu8~cloud2



*YY Inc. is hiring openstack and python developers. Interested? Check
http://www.nsbeta.info/jobs <http://www.nsbeta.info/jobs>*

--
Thanks,
Yuanle


On Thu, Mar 6, 2014 at 7:42 AM, Eric Blake <eblake@redhat.com> wrote:
> On 03/05/2014 04:32 PM, Martin Pavlásek wrote:
> > Hi
> >
> > I tried to restrict usage of some running VM by cpu.shares (i.e. set
to
> > 10 from original 1024) on loaded system and it seem doesn't work
as I
> > expected... all running processes has same CPU usage (by htop) :-/
>
> How exactly are you changing the cgroup settings?  Are you trying to
> directly modify cgroup files (unsupported), or using the libvirt API
> (such as via virsh schedinfo)?
>
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>

On Thu, Mar 06, 2014 at 12:32:28AM +0100, Martin Pavlásek
wrote:
> Hi
> 
> I tried to restrict usage of some running VM by cpu.shares (i.e. set to
> 10 from original 1024) on loaded system and it seem doesn't work as I
> expected... all running processes has same CPU usage (by htop) :-/
> Does anyone has same experience?
The cpu.shares variable doesn't provide absolute restriction on
CPU usage, rather it is doing relative prioritization.

eg if you have 2 cgroups that are at the same level in the hierarchy
and you give one shares=512 and one shared=1024, then the latter VM
will get twice the CPU scheduler time of the former. If the latter
VM is completely idle though, the former VM will not be capped in any
way.

If you want absolute caps then you need to use period/quota settings.
Also you want todo this via the   virsh schedinfo   command, not accessing
cgroups directly.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On Thu 06 Mar 2014 20:16:45 CET, Martin Pavlásek wrote:
> On 06/03/14 10:35, Daniel P. Berrange wrote:
>> On Thu, Mar 06, 2014 at 12:32:28AM +0100, Martin Pavlásek wrote:
>>> Hi
>>>
>>> I tried to restrict usage of some running VM by cpu.shares (i.e.
set to
>>> 10 from original 1024) on loaded system and it seem doesn't
work as I
>>> expected... all running processes has same CPU usage (by htop) :-/
>>> Does anyone has same experience?
>> The cpu.shares variable doesn't provide absolute restriction on
>> CPU usage, rather it is doing relative prioritization.
>>
>> eg if you have 2 cgroups that are at the same level in the hierarchy
>> and you give one shares=512 and one shared=1024, then the latter VM
>> will get twice the CPU scheduler time of the former. If the latter
>> VM is completely idle though, the former VM will not be capped in any
>> way.
>>
>> If you want absolute caps then you need to use period/quota settings.
>> Also you want todo this via the   virsh schedinfo   command, not
accessing
>> cgroups directly.
>>
>> Regards,
>> Daniel
> Thanks Daniel for reply, nice example!
> In my case I've got quite weird results
>
> guest: dd if=/dev/zero of=/dev/null for simulating heavy load
> host: dd if=... same as guest (to make host also loaded)
>
> host: htop for watching CPU usage (host has only one core), qemu (one
> VM, two threads) + dd (host) = 99% usage that is correct, as expected
> host: virsh schedinfo --set cpu_shares=10 <domain-ID>
> ...but whole load is still distributed equally - still ~30% for each.
>
> All processes are under itself default cgroup, so / for host dd and each
> VM in their own (something like /machine/<domain-name>). I didn't
change
> any cgroup directly in this case.
> After that qemu processes should shoconsume significantly less cpu than
> others, isn't it (change "relative priority" from 1024 to
10)?
> Limiting by period or quota works, but I would be really happy to use
> just cpu.shares - I don't want to write own process scheduler, weights
> of cpu sharing is sufficient for me.
>
> Do you have idea why this behaviour happen?
> Regards
>
I just find out what was wrong... :-) cpu.shares is applied only to 
process in same cgroup, so all common processes belong to / (cgroup 
root), all VMs under /machine/. So dd running on host belongs to /, but 
when I set cpu.shares of any VM, it wouldn't apply as I expected, 
because these processes are exists in different cgroups.

PS: I'm sorry Danied for answer only to you, I've mismatched Reply and 
Reply to all...

--
--
Martin Pavlásek <mpavlase@redhat.com>
 OpenStack QA Associate/Red Hat Czech/BRQ
 irc: mpavlase