libvirt users - Feb 2020 - libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params)

Hi everyone,
I have a question that you may be able to help me with. I had a live block
migration of a qemu-kvm guest fail (initiated via nova), in which the guest
remained running on the source, but if I try to re-initiate the live migration
it returns libvirtError: Timed out during operation: cannot acquire state change
lock (held by monitor=remoteDispatchDomainMigratePerform3Params). Looking at
blockjob --info I see there is a block copy that has been stuck at 23 % for a
few hours now. Is it safe to issue a —abort to the blockjob without impacting
the vm and will that resolve the lock?

Source:
libvirt-4.5.0-23.el7_7.5.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.7.1.x86_64

Destination:
libvirt-4.5.0-23.el7_7.5.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.7.1.x86_64

Thanks,
Blake

On Thu, Feb 27, 2020 at 18:28:44 -0700, Blake Anderson
wrote:
> 
> Hi everyone,
> I have a question that you may be able to help me with. I had a live block
migration of a qemu-kvm guest fail (initiated via nova), in which the guest
remained running on the source, but if I try to re-initiate the live migration
it returns libvirtError: Timed out during operation: cannot acquire state change
lock (held by monitor=remoteDispatchDomainMigratePerform3Params). Looking at
blockjob --info I see there is a block copy that has been stuck at 23 % for a
few hours now. Is it safe to issue a —abort to the blockjob without impacting
the vm and will that resolve the lock?
The domain job lock is held by the whole migration API not just the
blockjob. Aborting the blockjob or the migration thus should have the
same results after everything cleans up since the failed blockjob should
trigger abort of the whole migration anyways. So either of these should
do the trick:

virsh blockjob --abort $VM $DISK

virsh domjobabort $VM

Aborting a migration is safe, everything should just continue running at
the source.

Hey Peter, 
Thanks for the advice. It looks like neither of these methods worked. 
virsh block job —abort $domain $disk returned 
error: Timed out during operation: cannot acquire state change lock (held by
monitor=remoteDispatchDomainMigratePerform3Params)

I checked virsh domjobinfo $domain prior to attempted the domjobabort and it
just lists
Job type:         Cancelled

The domjobabort returned no output. 

Do you have any ideas of how to clear the lock short of restarting the qemu
process?


> On Feb 27, 2020, at 11:49 PM, Peter Krempa <pkrempa@redhat.com>
wrote:
> 
> On Thu, Feb 27, 2020 at 18:28:44 -0700, Blake Anderson wrote:
>> 
>> Hi everyone,
>> I have a question that you may be able to help me with. I had a live
block migration of a qemu-kvm guest fail (initiated via nova), in which the
guest remained running on the source, but if I try to re-initiate the live
migration it returns libvirtError: Timed out during operation: cannot acquire
state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params).
Looking at blockjob --info I see there is a block copy that has been stuck at 23
% for a few hours now. Is it safe to issue a —abort to the blockjob without
impacting the vm and will that resolve the lock?
> 
> The domain job lock is held by the whole migration API not just the
> blockjob. Aborting the blockjob or the migration thus should have the
> same results after everything cleans up since the failed blockjob should
> trigger abort of the whole migration anyways. So either of these should
> do the trick:
> 
> virsh blockjob --abort $VM $DISK
> 
> virsh domjobabort $VM
> 
> Aborting a migration is safe, everything should just continue running at
> the source.
>