libvirt users - Sep 2019 - virsh -c lxc:/// setvcpus and <vcpu> configuration fails

Hi folks!

i created a server with this XML file:

<domain type='lxc'>
  <name>lxctest1</name>
  <uuid>227bd347-dd1d-4bfd-81e1-01052e91ffe2</uuid>
  <metadata>
    <libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://centos.org/centos/6.9"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>
 <vcpu>2</vcpu>
  <numatune>
    <memory mode='strict' placement='auto'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <idmap>
    <uid start='0' target='200000' count='65535'/>
    <gid start='0' target='200000' count='65535'/>
  </idmap>
  <features>
    <privnet/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='mapped'>
      <source dir='/mnt'/>
      <target dir='/'/>
    </filesystem>
    <interface type='network'>
      <mac address='00:16:3e:3e:3e:bb'/>
      <source network='Public Network'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>


I would expect it to have 2 cpu cores and 1 GB RAM.


The RAM config works.
The CPU config does not:


[root@lxctest1 ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
Stepping:              4
CPU MHz:               2399.950
BogoMIPS:              4205.88
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23


It gives me all CPU's from the host.

I also tried it with

<cpu>
  <topology sockets='1' cores='2' threads='1'/>
</cpu>

That didnt help too.

I tried to modify the vcpus through virsh:



#virsh -c lxc:/// setvcpus lxctest1 2

error: this function is not supported by the connection driver:
virDomainSetVcpus

Which didnt work too.


This happens on:

Centos 7
Kernel: 5.1.15-1.el7.elrepo.x86_64

#virsh -V
Virsh command line tool of libvirt 4.5.0
See web site at https://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM LXC ESX Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Gluster ZFS
 Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline


and also on

Fedora 30
Kernel: 5.2.9-200.fc30.x86_64

# virsh -V
Virsh command line tool of libvirt 5.1.0
See web site at https://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM LXC LibXL OpenVZ VMware PHYP VirtualBox ESX
Hyper-V Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Sheepdog
Gluster ZFS
 Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline


-----------

Can anyone please tell me what i am doing wrong here ?

Thank you !

Greetings
Oliver

On Sun, Sep 15, 2019 at 12:21:08PM +0200, info@layer7.net
wrote:
>Hi folks!
>
>i created a server with this XML file:
>
><domain type='lxc'>
>  <name>lxctest1</name>
>  <uuid>227bd347-dd1d-4bfd-81e1-01052e91ffe2</uuid>
>  <metadata>
>    <libosinfo:libosinfo
>xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
>      <libosinfo:os id="http://centos.org/centos/6.9"/>
>    </libosinfo:libosinfo>
>  </metadata>
>  <memory unit='KiB'>1024000</memory>
>  <currentMemory unit='KiB'>1024000</currentMemory>
> <vcpu>2</vcpu>
>  <numatune>
>    <memory mode='strict' placement='auto'/>
>  </numatune>
>  <resource>
>    <partition>/machine</partition>
>  </resource>
>  <os>
>    <type arch='x86_64'>exe</type>
>    <init>/sbin/init</init>
>  </os>
>  <idmap>
>    <uid start='0' target='200000'
count='65535'/>
>    <gid start='0' target='200000'
count='65535'/>
>  </idmap>
>  <features>
>    <privnet/>
>  </features>
>  <clock offset='utc'/>
>  <on_poweroff>destroy</on_poweroff>
>  <on_reboot>restart</on_reboot>
>  <on_crash>destroy</on_crash>
>  <devices>
>    <emulator>/usr/libexec/libvirt_lxc</emulator>
>    <filesystem type='mount' accessmode='mapped'>
>      <source dir='/mnt'/>
>      <target dir='/'/>
>    </filesystem>
>    <interface type='network'>
>      <mac address='00:16:3e:3e:3e:bb'/>
>      <source network='Public Network'/>
>    </interface>
>    <console type='pty'>
>      <target type='lxc' port='0'/>
>    </console>
>  </devices>
></domain>
>
>
>I would expect it to have 2 cpu cores and 1 GB RAM.
>
>
>The RAM config works.
>The CPU config does not:
>
You probably checked /proc/meminfo.  That is provided by libvirt using fuse
filesystem, but at least it is guaranteed thanks to cgroups.  We do not (and I
don't think we even can, at least reliably) do that with cpuinfo.

[...]
>It gives me all CPU's from the host.
>
Although if you ran some perf benchmark it should just cap at 2 cpus.
>I also tried it with
>
><cpu>
>  <topology sockets='1' cores='2'
threads='1'/>
></cpu>
>
We should not allow this, IMO.  The reason is that we cannot guarantee or even
emulate this (or even the number of CPUs for that matter).  That's not how
containers work.  We can provide /proc/cpuinfo through a fuse filesystem, but if
the code actually asks the cpu directly there is no layer in which to emulate
the returned information.
>That didnt help too.
>
>I tried to modify the vcpus through virsh:
>
>
>
>#virsh -c lxc:/// setvcpus lxctest1 2
>
>error: this function is not supported by the connection driver:
>virDomainSetVcpus
>
This should not work for LXC, but it does not make sence because if you look at
the XML we allow `<vcpus>2</vcpus>`.
>Which didnt work too.
>
>
>This happens on:
>
Unfortunately anywhere, for the reasons said above.  Ideally it should not be
able to specify <vcpus/>, but rather just <cputune/>, but I
don't think we can
change that semantics now that we supported the former for quite some time.

Hi Martin,

thank you very much for your response!

Answers inline :)

>> Although if you ran some perf benchmark it should just cap at 2 cpus.
A cpuset.cpus from /sys/fs/cgroup.... will show with libvirtd: 0-23

So with 0-23 all 24 Cores will be assigned, AND useable.

I installed stress and run it with --cpu 8 or 16 or what ever. The
container receives, just according to the cgroup, all 24 cores without
any limitation.

/proc/cpuinfo
lscpu

it does not matter what you ask, all cpu's are given. Nothing is limited.

Also you will see the cpu load of the system, so also the accounting of
CPU is given to the container ( the container will see, that there is
load on the cpu ).


Compared to this,

a cpuset.cpus in lxc version 3 will show: 12,17,21,28 ( proxmox )

So 4 cores will be displayed, and as long as you dont put load on them,
you will see ~100% idle on all 4 cores.


Same goes for the implementation with lxd ( tested 3.x ).


And yes, sorry, i just saw later, that virDomainSetVcpus is simply not
supporting lxc. My fault, didnt read the documentation fully at that point.

------

So the general question is what is the expected result, when using
libvirt ( and i would really love to use libvirt, as we already use it
with kvm ) and would also like to use it with docker.

So the goal would be, to have something like what lxd / lxc is giving.

A container, that has N virtual CPUs, where only N virtual CPU's are
seen by the container through /proc/cpuinfo with its own cpu accounting.

So the question is:

Can libvirt set inside cgroups cpuset and cpuacct (correctly/at all) ?


The point here is the following:

If the software will check the container and will see on one hand, that
there are 24 CPU's available, but on the other hand only what ever else
is useable, things will go down a very dark road for us. And i guess not
only for us.

So its mandatory that cgroups are set correctly, at least in terms of
cpuset.

Can libvirt do this ? Or, as it seems now, can libvirt only manage KVM's
cgroups settings correctly ?

We just search for a better implementation for the workload. So
containers would be much better because of less overhead.

Again, thank you for your time !


-- 
Mit freundlichen Gruessen / Best regards

Oliver

Am 16.09.19 um 10:58 schrieb Martin Kletzander:
> On Sun, Sep 15, 2019 at 12:21:08PM +0200, info@layer7.net wrote:
>> Hi folks!
>>
>> i created a server with this XML file:
>>
>> <domain type='lxc'>
>>  <name>lxctest1</name>
>>  <uuid>227bd347-dd1d-4bfd-81e1-01052e91ffe2</uuid>
>>  <metadata>
>>    <libosinfo:libosinfo
>>
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
>>      <libosinfo:os id="http://centos.org/centos/6.9"/>
>>    </libosinfo:libosinfo>
>>  </metadata>
>>  <memory unit='KiB'>1024000</memory>
>>  <currentMemory unit='KiB'>1024000</currentMemory>
>> <vcpu>2</vcpu>
>>  <numatune>
>>    <memory mode='strict' placement='auto'/>
>>  </numatune>
>>  <resource>
>>    <partition>/machine</partition>
>>  </resource>
>>  <os>
>>    <type arch='x86_64'>exe</type>
>>    <init>/sbin/init</init>
>>  </os>
>>  <idmap>
>>    <uid start='0' target='200000'
count='65535'/>
>>    <gid start='0' target='200000'
count='65535'/>
>>  </idmap>
>>  <features>
>>    <privnet/>
>>  </features>
>>  <clock offset='utc'/>
>>  <on_poweroff>destroy</on_poweroff>
>>  <on_reboot>restart</on_reboot>
>>  <on_crash>destroy</on_crash>
>>  <devices>
>>    <emulator>/usr/libexec/libvirt_lxc</emulator>
>>    <filesystem type='mount' accessmode='mapped'>
>>      <source dir='/mnt'/>
>>      <target dir='/'/>
>>    </filesystem>
>>    <interface type='network'>
>>      <mac address='00:16:3e:3e:3e:bb'/>
>>      <source network='Public Network'/>
>>    </interface>
>>    <console type='pty'>
>>      <target type='lxc' port='0'/>
>>    </console>
>>  </devices>
>> </domain>
>>
>>
>> I would expect it to have 2 cpu cores and 1 GB RAM.
>>
>>
>> The RAM config works.
>> The CPU config does not:
>>
> 
> You probably checked /proc/meminfo.  That is provided by libvirt using fuse
> filesystem, but at least it is guaranteed thanks to cgroups.  We do not
> (and I
> don't think we even can, at least reliably) do that with cpuinfo.
> 
> [...]
> 
>> It gives me all CPU's from the host.
>>
> 
> Although if you ran some perf benchmark it should just cap at 2 cpus.
> 
>> I also tried it with
>>
>> <cpu>
>>  <topology sockets='1' cores='2'
threads='1'/>
>> </cpu>
>>
> 
> We should not allow this, IMO.  The reason is that we cannot guarantee
> or even
> emulate this (or even the number of CPUs for that matter).  That's not
how
> containers work.  We can provide /proc/cpuinfo through a fuse
> filesystem, but if
> the code actually asks the cpu directly there is no layer in which to
> emulate
> the returned information.
> 
>> That didnt help too.
>>
>> I tried to modify the vcpus through virsh:
>>
>>
>>
>> #virsh -c lxc:/// setvcpus lxctest1 2
>>
>> error: this function is not supported by the connection driver:
>> virDomainSetVcpus
>>
> 
> This should not work for LXC, but it does not make sence because if you
> look at
> the XML we allow `<vcpus>2</vcpus>`.
> 
>> Which didnt work too.
>>
>>
>> This happens on:
>>
> 
> Unfortunately anywhere, for the reasons said above.  Ideally it should
> not be
> able to specify <vcpus/>, but rather just <cputune/>, but I
don't think
> we can
> change that semantics now that we supported the former for quite some time.

Hi,

ok i tried:

  <vcpu placement='static'>2</vcpu>
  <iothreads>2</iothreads>
  <iothreadids>
    <iothread id='1'/>
    <iothread id='2'/>
  </iothreadids>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <emulatorpin cpuset='0-1'/>
    <iothreadpin iothread='1' cpuset='0'/>
    <iothreadpin iothread='2' cpuset='1'/>
  </cputune>

aswell as

  <vcpu placement='static' cpuset='0-1'
current='2'>4</vcpu>


and, it shows in the cpuset cgroup:

#cat cpuset.cpus
0-1

# cat cpuset.effective_cpus
0-1

And yes, the CPU power is reduced to two cores.

But still, /proc/cpuinfo will show _all_ cpu cores of the hostmachine.

Also the total cpu usage / load, if queried, will be the cpu usage /
load of the hostmachine.

That confuse the applications as they monitor the cpu load/count and try
to balance out things.

Is there really no way to tell libvirt to create the cgroups in a way to
just show X cpu's to the system ?

When i run purely lxc or lxd, its no problem. But i would like to handle
things through libvirt because also the qemu-kvm stuff is handled
through libvirt.

Any ideas ? I also take dark hacking stuff :)

Greetings
Oliver






Am 16.09.19 um 10:58 schrieb Martin Kletzander:
> On Sun, Sep 15, 2019 at 12:21:08PM +0200, info@layer7.net wrote:
>> Hi folks!
>>
>> i created a server with this XML file:
>>
>> <domain type='lxc'>
>>  <name>lxctest1</name>
>>  <uuid>227bd347-dd1d-4bfd-81e1-01052e91ffe2</uuid>
>>  <metadata>
>>    <libosinfo:libosinfo
>>
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
>>      <libosinfo:os id="http://centos.org/centos/6.9"/>
>>    </libosinfo:libosinfo>
>>  </metadata>
>>  <memory unit='KiB'>1024000</memory>
>>  <currentMemory unit='KiB'>1024000</currentMemory>
>> <vcpu>2</vcpu>
>>  <numatune>
>>    <memory mode='strict' placement='auto'/>
>>  </numatune>
>>  <resource>
>>    <partition>/machine</partition>
>>  </resource>
>>  <os>
>>    <type arch='x86_64'>exe</type>
>>    <init>/sbin/init</init>
>>  </os>
>>  <idmap>
>>    <uid start='0' target='200000'
count='65535'/>
>>    <gid start='0' target='200000'
count='65535'/>
>>  </idmap>
>>  <features>
>>    <privnet/>
>>  </features>
>>  <clock offset='utc'/>
>>  <on_poweroff>destroy</on_poweroff>
>>  <on_reboot>restart</on_reboot>
>>  <on_crash>destroy</on_crash>
>>  <devices>
>>    <emulator>/usr/libexec/libvirt_lxc</emulator>
>>    <filesystem type='mount' accessmode='mapped'>
>>      <source dir='/mnt'/>
>>      <target dir='/'/>
>>    </filesystem>
>>    <interface type='network'>
>>      <mac address='00:16:3e:3e:3e:bb'/>
>>      <source network='Public Network'/>
>>    </interface>
>>    <console type='pty'>
>>      <target type='lxc' port='0'/>
>>    </console>
>>  </devices>
>> </domain>
>>
>>
>> I would expect it to have 2 cpu cores and 1 GB RAM.
>>
>>
>> The RAM config works.
>> The CPU config does not:
>>
> 
> You probably checked /proc/meminfo.  That is provided by libvirt using fuse
> filesystem, but at least it is guaranteed thanks to cgroups.  We do not
> (and I
> don't think we even can, at least reliably) do that with cpuinfo.
> 
> [...]
> 
>> It gives me all CPU's from the host.
>>
> 
> Although if you ran some perf benchmark it should just cap at 2 cpus.
> 
>> I also tried it with
>>
>> <cpu>
>>  <topology sockets='1' cores='2'
threads='1'/>
>> </cpu>
>>
> 
> We should not allow this, IMO.  The reason is that we cannot guarantee
> or even
> emulate this (or even the number of CPUs for that matter).  That's not
how
> containers work.  We can provide /proc/cpuinfo through a fuse
> filesystem, but if
> the code actually asks the cpu directly there is no layer in which to
> emulate
> the returned information.
> 
>> That didnt help too.
>>
>> I tried to modify the vcpus through virsh:
>>
>>
>>
>> #virsh -c lxc:/// setvcpus lxctest1 2
>>
>> error: this function is not supported by the connection driver:
>> virDomainSetVcpus
>>
> 
> This should not work for LXC, but it does not make sence because if you
> look at
> the XML we allow `<vcpus>2</vcpus>`.
> 
>> Which didnt work too.
>>
>>
>> This happens on:
>>
> 
> Unfortunately anywhere, for the reasons said above.  Ideally it should
> not be
> able to specify <vcpus/>, but rather just <cputune/>, but I
don't think
> we can
> change that semantics now that we supported the former for quite some time.