libvirt users - Jan 2019 - libvirt 5.0.0 - LXC container still in "virsh list" output after shutdown

Hello.

Centos 7.6 with libvirt build from base "virt" repository:
libvirt-daemon-driver-lxc-5.0.0-1.el7.x86_64
libvirt-client-5.0.0-1.el7.x86_64
libvirt-daemon-5.0.0-1.el7.x86_64
libvirt-daemon-driver-network-5.0.0-1.el7.x86_64
libvirt-libs-5.0.0-1.el7.x86_64
+
systemd-219-62.el7_6.2.x86_64

Now lxc containers with  type='direct' can be started,  but can't be
stopped :)
Before shutdown.
# virsh  list --all
 Id      Name                   State
------------------------------------------
 16312   test.lxc               running

# machinectl | grep test
lxc-16312-test.lxc            container libvirt-lxc

# pstree -apA  16312
libvirt_lxc,16312 --name test.lxc --console 42 --security=none
--handshake 45 --veth macvlan0
  |-systemd,16315
  |   |-agetty,16690 --noclear --keep-baud console 115200 38400 9600 vt220
  |   |-dbus-daemon,16659 --system --address=systemd: --nofork
--nopidfile --systemd-activation
  |   |-rsyslogd,17614 -n
  |   |   |-{rsyslogd},17616
  |   |   `-{rsyslogd},17626
  |   |-sshd,17613 -D
  |   |-systemd-journal,16613
  |   `-systemd-logind,16657
  `-{libvirt_lxc},16319


Shutdown.
# virsh  shutdown 16312
Domain 16312 is being shutdown
# echo $?
0

After.
In list output still present:
# virsh  list --all
 Id      Name                   State
------------------------------------------
 16312   test.lxc               running
# machinectl | grep test | wc -l
0

No main process:
# ps axuwwf | grep libvirt_lxc | grep test | wc -l
0

LXC realy stopped, but only virsh still show it in list
Cgroups "blkio, cpuacct, memory, cpu, cpu,cpuacct, devices, hugetlb,
pids" deleted.
But another cgroups still present in fs:
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/cpuset/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/freezer/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/net_cls/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/net_cls,net_prio/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/net_prio/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope
drwxr-xr-x 2 root root 0 Jan 21 13:07
/sys/fs/cgroup/perf_event/machine.slice/machine-lxc\x2d16312\x2dtest.lxc.scope

Restarted libvirtd temporary place container in correct state:
# systemctl restart libvirtd
# virsh list --all | grep test
 -       test.lxc            shut off

libvirt 4.5.0 and 4.10.0 perform correct LXC shutdown.

b.r.
  Maxim Kozin

> Centos 7.6 with libvirt build from base "virt" repository:
> libvirt-daemon-driver-lxc-5.0.0-1.el7.x86_64
> libvirt-client-5.0.0-1.el7.x86_64
> libvirt-daemon-5.0.0-1.el7.x86_64
> libvirt-daemon-driver-network-5.0.0-1.el7.x86_64
> libvirt-libs-5.0.0-1.el7.x86_64
> +
> systemd-219-62.el7_6.2.x86_64
>
> Now lxc containers with  type='direct' can be started,  but
can't be stopped
Additional info.
When shutdown LZXC container with --mode signal, nothing happened on
virsh console LXC_NAME
If use --mode initctl, container powering off:
...
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Remote File Systems.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped target Local File Systems.
         Unmounting /proc/sys/net/ipv4...
[  OK  ] Stopped target Local File Systems (Pre).
         Unmounting /proc/sys/net/ipv6...
         Unmounting /dev/tty1...
         Unmounting /proc/meminfo...
         Unmounting /dev/ptmx...
[  OK  ] Stopped Configure read-only root support.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[  OK  ] Reached target Final Step.
         Starting Power-Off...
and virsh console disconnected.
But as reported before, virsh list show container in status running.
Interesting, that in test python script for such stucked container we
get error on call domain.info():
(Pdb)  self.state_id, self.maxmem, self.mem, self.cpus, self.cput
self.domain.info()
*** libvirtError: operation failed: Cannot read cputime for domain

where self.domain is correct domain pointer:
(Pdb) self.domain.__dict__
{'_conn': <libvirt.virConnect object at 0x7ff95f881550>,
'_o':
<capsule object "virDomainPtr" at 0x7ff95f8cca80>}

Compare with case just after libvirtd restart and domain stopped:
(Pdb) self.domain.info()
[5, 2097152L, 2097152L, 2, 0L]

Problem reproduced on 2 nodes for lxc. Qemu container on same nodes
shutdown correct.
b.r.
 Maxim Kozin

https://bugzilla.redhat.com/show_bug.cgi?id=1681180

b.r.
  Maxim Kozin