libvirt users - Aug 2019 - [libvirtd] qemu_process: reset CPU affinity to all enabled CPUs, when runs in custom cpuset

Hello All,

Since 4.5.0-23.el7 version (Red Hat 7.7), when I launch pinned VM,
libvirtd reset CPU affinity to all enabled in host CPUs, if it runs in
custom cpuset.
I can't reproduce this behavior with 4.5.0-10.el7_6.12 with the same
kernel version (Red Hat 7.7).

Libvirt runs in a custom cpuset 'libvirt', where the number of
available cpus is restricted to 0,2,4,6,8.
And this 'libvirt' cpuset is created in a system with total cpus
number: 40 (all cpus are enabled in BIOS) and I have '0-39' range in
/sys/fs/cgroup/cpuset/cpuset.cpus.

When a VM with pinned vcpus is launched, VM XML config is in attachement:

  <vcpu placement='static'>2</vcpu>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='6'/>

I have the following error:

# virsh create /tmp/vm1_2vms-2cores_host1.xml
error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied

And in the debug log I can see, that
/sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset
list: 0,2,4,6,8.
This list at the beginning was successfully inherited and set in
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus.
But finally libvirtd tries to reset inherited cpus and set there:
"0-39"

2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 :
Make group
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug :
virCgroupCpuSetInherit:989 : Setting up inheritance
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope ->
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.357+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus'
to '0,2,4,6,8'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems'
to '0-1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate'
to '1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller memory
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller devices
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller freezer
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller blkio
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller net_cls
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller perf_event
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 :
Not creating systemd controller group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 :
Done making controllers for group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus'
to '0-39'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806
: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied

The same log snippet with 4.5.0-10.el7_6.12 version, everything works well:

2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 :
Make group
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:989 : Setting up inheritance
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope ->
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus'
to '0,2,4,6,8'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems'
to '0-1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate'
to '1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller memory
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller devices
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller freezer
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller blkio
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller net_cls
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller perf_event
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 :
Not creating systemd controller group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 :
Done making controllers for group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems'
to '0'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks'
to '27045'

Would someone give me a hint, please, if we need to change settings
lied to NUMA or vCPUs in VM XML config according to last changed in
libvirtd ?

Please, find a bug report about this issue here:
https://bugzilla.redhat.com/show_bug.cgi?id=1746517

I suppose, that this may be an impact of one of the following patches
applied to 4.5.0-10.el7_6.12 source code version:

libvirt-qemu-Rework-setting-process-affinity.patch
libvirt-qemu-Fix-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-NULL-pointer-access-in-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-leak-in-qemuProcessInitCpuAffinity.patch

How reproducible: Always

Steps to Reproduce:
1. create a cpuset with reduced cpus list and launch libvirtd in it
2. prepare VM XML with pinned vcpus and <numatune> setting
3. virsh create vm.xml

Actual results:

error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied

Expected results:
should work as with a previous version 4.5.0-10.el7_6.12.

Additional info:

# ps auxf | grep libvirt
root     24571  2.3  0.0 1895564 28052 ?       Ssl  16:52   0:00
/usr/sbin/libvirtd

# grep Cpus_allowed /proc/24571/task/*/status
/proc/24571/task/24571/status:Cpus_allowed:     00,00000554
/proc/24571/task/24571/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24572/status:Cpus_allowed:     00,00000554
/proc/24571/task/24572/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24573/status:Cpus_allowed:     00,00000554
/proc/24571/task/24573/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24574/status:Cpus_allowed:     00,00000554
/proc/24571/task/24574/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24575/status:Cpus_allowed:     00,00000554
..
for all threads

Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"

        Sockets: 2, Cores: 20, HyperThreads: 40

         socket 0                 socket 1
  +---------------------+  +---------------------+
  |        c0        c1 |  |        c0        c1 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  0| 20| |  2| 22| |  | |  1| 21| |  3| 23| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c2        c3 |  |        c2        c3 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  4| 24| |  6| 26| |  | |  5| 25| |  7| 27| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c4        c8 |  |        c4        c8 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  8| 28| | 10| 30| |  | |  9| 29| | 11| 31| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c9       c10 |  |        c9       c10 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 12| 32| | 14| 34| |  | | 13| 33| | 15| 35| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |       c11       c12 |  |       c11       c12 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 16| 36| | 18| 38| |  | | 17| 37| | 19| 39| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  +---------------------+  +---------------------+

# cat  /sys/fs/cgroup/cpuset/cpuset.cpus
0-39

# cat  /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus
2,4,6,8,10

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus
0,2,4,6,8