libvirt users - Oct 2017 - virsh does no longer respond to commands

Hi,

after a recent live backup done with a snapshot, blockcommit failed and a
subsequent (manual) blockjob abort got stuck. Now we are unable to do basic
operations on the virtual machine like if it was frozen and when issuing
another command we see the following:

$ virsh shutdown $domain
error: Failed to shutdown domain $domain
error: Timed out during operation: cannot acquire state change lock (held
by remoteDispatchDomainBlockJobAbort)

$ virsh blockcommit $domain --active --pivot
error: Timed out during operation: cannot acquire state change lock (held
by remoteDispatchDomainBlockJobAbort)

$ virsh blockjob $domain --abort
error: Timed out during operation: cannot acquire state change lock (held
by remoteDispatchDomainBlockJobAbort)

Houston, we've got a problem. We have roughly 16 hours to fix this. Any
help is appreciated!
-- 
Dominik Psenner

Luckily, we were able to sort this problem out by rebooting the hosting 
machine and restoring the guest machine from a backup. Did we stumble 
upon a bug? "blockjob abort" should not leave behind orphaned locks
when
it completed, should it?


On 2017-10-22 14:04, Dominik Psenner wrote:
> Hi,
>
> after a recent live backup done with a snapshot, blockcommit failed and a
> subsequent (manual) blockjob abort got stuck. Now we are unable to do basic
> operations on the virtual machine like if it was frozen and when issuing
> another command we see the following:
>
> $ virsh shutdown $domain
> error: Failed to shutdown domain $domain
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
>
> $ virsh blockcommit $domain --active --pivot
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
>
> $ virsh blockjob $domain --abort
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
>
> Houston, we've got a problem. We have roughly 16 hours to fix this. Any
> help is appreciated!

On 10/22/2017 02:04 PM, Dominik Psenner wrote:
> Hi,
> 
> after a recent live backup done with a snapshot, blockcommit failed and a
> subsequent (manual) blockjob abort got stuck. Now we are unable to do basic
> operations on the virtual machine like if it was frozen and when issuing
> another command we see the following:
> 
> $ virsh shutdown $domain
> error: Failed to shutdown domain $domain
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
> 
> $ virsh blockcommit $domain --active --pivot
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
> 
> $ virsh blockjob $domain --abort
> error: Timed out during operation: cannot acquire state change lock (held
> by remoteDispatchDomainBlockJobAbort)
> 
This looks like some API forgot to unset the job before returning. In
that case, restarting libvirtd is the only option. If this happened on
the latest release please do file a bug. Otherwise try with the latest
release.

Michal

It has been the version available on ubuntu 16.04 lts:

$ virsh --version
1.3.1

Please file a bug for me if you think it is one.

On 25 Oct 2017 2:34 p.m., "Michal Privoznik"
<mprivozn@redhat.com> wrote:
> On 10/22/2017 02:04 PM, Dominik Psenner wrote:
> > Hi,
> >
> > after a recent live backup done with a snapshot, blockcommit failed
and a
> > subsequent (manual) blockjob abort got stuck. Now we are unable to do
> basic
> > operations on the virtual machine like if it was frozen and when
issuing
> > another command we see the following:
> >
> > $ virsh shutdown $domain
> > error: Failed to shutdown domain $domain
> > error: Timed out during operation: cannot acquire state change lock
(held
> > by remoteDispatchDomainBlockJobAbort)
> >
> > $ virsh blockcommit $domain --active --pivot
> > error: Timed out during operation: cannot acquire state change lock
(held
> > by remoteDispatchDomainBlockJobAbort)
> >
> > $ virsh blockjob $domain --abort
> > error: Timed out during operation: cannot acquire state change lock
(held
> > by remoteDispatchDomainBlockJobAbort)
> >
>
> This looks like some API forgot to unset the job before returning. In
> that case, restarting libvirtd is the only option. If this happened on
> the latest release please do file a bug. Otherwise try with the latest
> release.
>
> Michal
>
>