libvirt users - Jul 2016 - Virtual machine in state "in shutdown"

Hello,

We are currently facing a strange situation. One of our VM is shown by
'virsh list' as in state "in shutdown" but there is no more a
qemu-kvm
process linked to it.

So we have a few questions:

1. What does means the state in shutdown (I have not found much information
about it)?
2. How to cleanly "shutdown" the vm, or more correctly, clean the
status in
virsh/libvirt?


Regards,


Roland Everaert.

On Tue, Jul 05, 2016 at 02:50:25PM +0200, Roland Everaert
wrote:
> Hello,
> 
> We are currently facing a strange situation. One of our VM is shown by
> 'virsh list' as in state "in shutdown" but there is no
more a qemu-kvm
> process linked to it.
> 
> So we have a few questions:
> 
> 1. What does means the state in shutdown (I have not found much information
> about it)?
'in shutdown' means QEMU has gone away, and libvirt is cleaning up
any state it has left over. There's little info on it, since you
should almost never seee it - a VM is only in that state for a tiny
fraction of a second normally.
> 2. How to cleanly "shutdown" the vm, or more correctly, clean the
status in
> virsh/libvirt?
If you have a "in shutdown" state that persists, it is probably a sign
of a bug in libvirt. You could see if libvirtd logged any interesting
error messages. As for cleaning the state, you can probably achieve
that by simply restarting libvirtd - it redetects VM state on startup

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Restarting libvirtd doesn't change the situation.

But looking into the logs I see the following:

- Last lines in /var/log/libvirt/libvirtd.log:

2016-07-05 13:26:42.792+0000: 24552: warning : qemuProcessKill:4419 : Timed
out waiting after SIGKILL to process 48301
2016-07-05 13:26:42.792+0000: 24552: error : qemuDomainDestroyFlags:2120 :
operation failed: failed to kill qemu process with SIGTERM

- lookup for process 48301:

[root@lpextvms003c ~]# ps -ef | grep 48301
root      1114 24243  0 16:02 pts/1    00:00:00 grep 48301
oneadmin 48301     1  4 Apr18 ?        3-08:14:18 [qemu-kvm] <defunct>


The name of the strange VM is one-357
I have seen that the last log in /var/log/libvirt/qemu/one-357.log are
dated 2016/04/18.

- Info on process one-357:

[root@lpextvms003c ~]# cat /var/run/libvirt/qemu/one-357.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
  virsh edit one-357
or other application using the libvirt API.
-->

<domstatus state='shutdown' reason='unknown'
pid='48301'>
  <monitor path='/var/lib/libvirt/qemu/one-357.monitor'
json='1'
type='unix'/>
  <vcpus>
    <vcpu pid='48319'/>
    <vcpu pid='48320'/>
  </vcpus>


So the process is defunct, does a 'kill -9 48301' could fix the problem
with a restart of libvirtd?


Thanks,

Roland.


On Tue, Jul 5, 2016 at 2:55 PM, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Tue, Jul 05, 2016 at 02:50:25PM +0200, Roland Everaert wrote:
> > Hello,
> >
> > We are currently facing a strange situation. One of our VM is shown by
> > 'virsh list' as in state "in shutdown" but there is
no more a qemu-kvm
> > process linked to it.
> >
> > So we have a few questions:
> >
> > 1. What does means the state in shutdown (I have not found much
> information
> > about it)?
>
> 'in shutdown' means QEMU has gone away, and libvirt is cleaning up
> any state it has left over. There's little info on it, since you
> should almost never seee it - a VM is only in that state for a tiny
> fraction of a second normally.
>
> > 2. How to cleanly "shutdown" the vm, or more correctly,
clean the status
> in
> > virsh/libvirt?
>
> If you have a "in shutdown" state that persists, it is probably a
sign
> of a bug in libvirt. You could see if libvirtd logged any interesting
> error messages. As for cleaning the state, you can probably achieve
> that by simply restarting libvirtd - it redetects VM state on startup
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>