Roman Mohr
2018-Sep-11 12:00 UTC
[libvirt-users] Running libvirtd outside of a shell results in qemu defunct processes
Hi, In KubeVirt we are right now starting libvirt via a wrapper shell script. I want to remove that wrapper and directly start libvirtd from our golang code. This means that our golang code is PID 1 in the container and we start libvirtd as a fork from there. When I did this, strange things started to happen. Libvirt was still able to start and manage qemu processes, but stopping the qemu processes stopped working. I can see that all qemu processes, the ones with VMs and the ones used by libvirt to do some host capability detection, stay in the process table as zombies: qemu 54 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-i38] <defunct> qemu 56 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-i38] <defunct> qemu 61 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-i38] <defunct> qemu 63 0.1 0.0 0 0 ? Z 11:44 0:00 [qemu-system-i38] <defunct> qemu 70 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 72 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 79 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 81 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 86 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 88 0.3 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 93 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 95 0.1 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> So to sum up, when I start our golang wrapper around libvirtd inside a bash script without "exec" everything works as usual. If I start in the container pid namespace the golang wrapper directly, or use "exec" from the bash script, things stop working. I also see the libvirt repeatedly tries to kill the qemu processes, but they are stuck. Maybe someone has a clue why things stop working. Thanks and Best Regards, Roman
Roman Mohr
2018-Sep-11 14:28 UTC
Re: [libvirt-users] Running libvirtd outside of a shell results in qemu defunct processes
On Tue, Sep 11, 2018 at 2:00 PM Roman Mohr <rmohr@redhat.com> wrote:> > Hi, > > In KubeVirt we are right now starting libvirt via a wrapper shell > script. I want to remove that wrapper and directly start libvirtd from > our golang code. This means that our golang code is PID 1 in the > container and we start libvirtd as a fork from there. > > When I did this, strange things started to happen. Libvirt was still > able to start and manage qemu processes, but stopping the qemu > processes stopped working. I can see that all qemu processes, the ones > with VMs and the ones used by libvirt to do some host capability > detection, stay in the process table as zombies: > > qemu 54 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-i38] <defunct> > qemu 56 0.2 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-i38] <defunct> > qemu 61 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-i38] <defunct> > qemu 63 0.1 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-i38] <defunct> > qemu 70 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 72 0.2 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 79 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 81 0.2 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 86 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 88 0.3 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 93 0.0 0.0 0 0 ? Zs 11:44 0:00 > [qemu-system-x86] <defunct> > qemu 95 0.1 0.0 0 0 ? Z 11:44 0:00 > [qemu-system-x86] <defunct> > > So to sum up, when I start our golang wrapper around libvirtd inside a > bash script without "exec" everything works as usual. If I start in > the container pid namespace the golang wrapper directly, or use "exec" > from the bash script, things stop working. I also see the libvirt > repeatedly tries to kill the qemu processes, but they are stuck.Never mind. The solution was pretty easy. Since my process is PID 1, all qemu processes were orphaned by libvirt and reassigned to my process. I am now listening for SIGCHILD and reap them by calling waitpid. Best Regards, Roman> > Maybe someone has a clue why things stop working. > > Thanks and Best Regards, > > Roman