libvirt users - Jul 2016 - Re: Virtual machine in state "in shutdown"

Restarting libvirtd doesn't change the situation.

But looking into the logs I see the following:

- Last lines in /var/log/libvirt/libvirtd.log:

2016-07-05 13:26:42.792+0000: 24552: warning : qemuProcessKill:4419 : Timed
out waiting after SIGKILL to process 48301
2016-07-05 13:26:42.792+0000: 24552: error : qemuDomainDestroyFlags:2120 :
operation failed: failed to kill qemu process with SIGTERM

- lookup for process 48301:

[root@lpextvms003c ~]# ps -ef | grep 48301
root      1114 24243  0 16:02 pts/1    00:00:00 grep 48301
oneadmin 48301     1  4 Apr18 ?        3-08:14:18 [qemu-kvm] <defunct>


The name of the strange VM is one-357
I have seen that the last log in /var/log/libvirt/qemu/one-357.log are
dated 2016/04/18.

- Info on process one-357:

[root@lpextvms003c ~]# cat /var/run/libvirt/qemu/one-357.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
  virsh edit one-357
or other application using the libvirt API.
-->

<domstatus state='shutdown' reason='unknown'
pid='48301'>
  <monitor path='/var/lib/libvirt/qemu/one-357.monitor'
json='1'
type='unix'/>
  <vcpus>
    <vcpu pid='48319'/>
    <vcpu pid='48320'/>
  </vcpus>


So the process is defunct, does a 'kill -9 48301' could fix the problem
with a restart of libvirtd?


Thanks,

Roland.


On Tue, Jul 5, 2016 at 2:55 PM, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Tue, Jul 05, 2016 at 02:50:25PM +0200, Roland Everaert wrote:
> > Hello,
> >
> > We are currently facing a strange situation. One of our VM is shown by
> > 'virsh list' as in state "in shutdown" but there is
no more a qemu-kvm
> > process linked to it.
> >
> > So we have a few questions:
> >
> > 1. What does means the state in shutdown (I have not found much
> information
> > about it)?
>
> 'in shutdown' means QEMU has gone away, and libvirt is cleaning up
> any state it has left over. There's little info on it, since you
> should almost never seee it - a VM is only in that state for a tiny
> fraction of a second normally.
>
> > 2. How to cleanly "shutdown" the vm, or more correctly,
clean the status
> in
> > virsh/libvirt?
>
> If you have a "in shutdown" state that persists, it is probably a
sign
> of a bug in libvirt. You could see if libvirtd logged any interesting
> error messages. As for cleaning the state, you can probably achieve
> that by simply restarting libvirtd - it redetects VM state on startup
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>

On Tue, Jul 05, 2016 at 04:20:58PM +0200, Roland Everaert
wrote:
> Restarting libvirtd doesn't change the situation.
> 
> But looking into the logs I see the following:
> 
> - Last lines in /var/log/libvirt/libvirtd.log:
> 
> 2016-07-05 13:26:42.792+0000: 24552: warning : qemuProcessKill:4419 : Timed
> out waiting after SIGKILL to process 48301
> 2016-07-05 13:26:42.792+0000: 24552: error : qemuDomainDestroyFlags:2120 :
> operation failed: failed to kill qemu process with SIGTERM
> 
> - lookup for process 48301:
> 
> [root@lpextvms003c ~]# ps -ef | grep 48301
> root      1114 24243  0 16:02 pts/1    00:00:00 grep 48301
> oneadmin 48301     1  4 Apr18 ?        3-08:14:18 [qemu-kvm]
<defunct>
Ok, so your QEMU process has not in fact died completely, so the
libvirt state is correct.

Usual cause of a qemu going into a defunct state is a storage
failure which causes QEMU to be stuck in an uninterruptable
I/O operation wait state.
> So the process is defunct, does a 'kill -9 48301' could fix the
problem
> with a restart of libvirtd?
No, as you see from the logs above, libvirt already tried to
send a SIGKILL and it had no effect. You need to figure out
what has caused QEMU to get stuck - most likely a storage I/O
problem on your machine

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

So, the disks of our VMs are stored on a GlusterFS volume.

It seems that the disk of the VM is locked, for some reason, by GlusterFS.
My colleague tried to unlock the file, but it completely lock the volume.
So he is currently restarting the host of the VMs and restart all the VMs.

But we think that there will be, still, a problème for the "locked"
VM, we
will see.


Thanks for your help,


Roland.



On Tue, Jul 5, 2016 at 4:24 PM, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Tue, Jul 05, 2016 at 04:20:58PM +0200, Roland Everaert wrote:
> > Restarting libvirtd doesn't change the situation.
> >
> > But looking into the logs I see the following:
> >
> > - Last lines in /var/log/libvirt/libvirtd.log:
> >
> > 2016-07-05 13:26:42.792+0000: 24552: warning : qemuProcessKill:4419 :
> Timed
> > out waiting after SIGKILL to process 48301
> > 2016-07-05 13:26:42.792+0000: 24552: error :
qemuDomainDestroyFlags:2120
> :
> > operation failed: failed to kill qemu process with SIGTERM
> >
> > - lookup for process 48301:
> >
> > [root@lpextvms003c ~]# ps -ef | grep 48301
> > root      1114 24243  0 16:02 pts/1    00:00:00 grep 48301
> > oneadmin 48301     1  4 Apr18 ?        3-08:14:18 [qemu-kvm]
<defunct>
>
> Ok, so your QEMU process has not in fact died completely, so the
> libvirt state is correct.
>
> Usual cause of a qemu going into a defunct state is a storage
> failure which causes QEMU to be stuck in an uninterruptable
> I/O operation wait state.
>
> > So the process is defunct, does a 'kill -9 48301' could fix
the problem
> > with a restart of libvirtd?
>
> No, as you see from the logs above, libvirt already tried to
> send a SIGKILL and it had no effect. You need to figure out
> what has caused QEMU to get stuck - most likely a storage I/O
> problem on your machine
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>