libvirt users - Dec 2016 - libvirtError: block copy still active: disk not ready for pivot yet

Hi guys,

When performing live merge, in few cases, we see the following exception:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 736, in
wrapper
    return f(*a, **kw)
  File "/usr/share/vdsm/virt/vm.py", line 5278, in run
    self.tryPivot()
  File "/usr/share/vdsm/virt/vm.py", line 5247, in tryPivot
    ret = self.vm._dom.blockJobAbort(self.drive.name, flags)
  File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 124, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in
wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 733, in
blockJobAbort
    if ret == -1: raise libvirtError ('virDomainBlockJobAbort()
failed', dom=self)
libvirtError: block copy still active: disk 'vdb' not ready for pivot
yet

That exception observed in following BZs:

https://bugzilla.redhat.com/1376580
https://bugzilla.redhat.com/1397122

I am trying to understand what this exception indicates in order to handle
it appropriately when thrown by libvirt.

Thanks,
Ala

On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino
wrote:
> Hi guys,
> 
> When performing live merge, in few cases, we see the following exception:
[...]
> libvirtError: block copy still active: disk 'vdb' not ready for
pivot yet
You can see if the block operation is still in progress or not by doing:
`sudo virsh blockjob vm1 vdb --info`.
> That exception observed in following BZs:
> 
> https://bugzilla.redhat.com/1376580
> https://bugzilla.redhat.com/1397122
[I see, that the above are oVirt bugs.]
> I am trying to understand what this exception indicates in order to handle
> it appropriately when thrown by libvirt.
Here's the commit that adds the above error message.  In short: Until
the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY, the
copy / mirror block operation is truly not complete.


    commit eae59247c59aa02147b2b4a50177e8e877fdb218
    Author: Peter Krempa <pkrempa at redhat.com>
    Date:   Wed Jul 15 15:11:02 2015 +0200
    
        qemu: Update state of block job to READY only if it actually is ready
        
        Few parts of the code looked at the current progress of and assumed that
        a two phase blockjob is in the _READY state as soon as the progress
        reached 100% (info.cur == info.end). In current versions of qemu this
        assumption is invalid and qemu exposes a new flag 'ready' in the
        query-block-jobs output that is set to true if the job is actually
        finished.
        
        This patch adds internal data handling for reading the 'ready'
flag and
        acting appropriately as long as the flag is present.
        
        While this still doesn't fix the virsh client problem with two phase
        block jobs and the --pivot option, it at least improves the error
        message:
        
        $ virsh blockcommit  --wait --verbose vm vda  --base vda[1] --active
--pivot
        Block commit: [100 %]error: failed to pivot job for disk vda
        error: internal error: unable to execute QEMU command
'block-job-complete': The active block job for device
'drive-virtio-disk0' cannot be completed
        
        to
        
        $ virsh blockcommit  --wait --verbose VM vda  --base vda[1] --active
--pivot
        Block commit: [100 %]error: failed to pivot job for disk vda
        error: block copy still active: disk 'vda' not ready for pivot
yet


[...]

-- 
/kashyap

I understand that libvirt provides async events that are triggered once the
block job reaches ready state and successfully pivots as it is reported by
qemu.
Whats the event that is triggered and what's the handler that we have to
implement?
To be more specific, I am looking for the onEVENT callback method that is
invoked when the event is fired.

On Tue, Jan 3, 2017 at 6:28 PM, Kashyap Chamarthy <kchamart@redhat.com>
wrote:
> On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino wrote:
> > Hi guys,
> >
> > When performing live merge, in few cases, we see the following
exception:
>
> [...]
>
> > libvirtError: block copy still active: disk 'vdb' not ready
for pivot yet
>
> You can see if the block operation is still in progress or not by doing:
> `sudo virsh blockjob vm1 vdb --info`.
>
> > That exception observed in following BZs:
> >
> > https://bugzilla.redhat.com/1376580
> > https://bugzilla.redhat.com/1397122
>
> [I see, that the above are oVirt bugs.]
>
> > I am trying to understand what this exception indicates in order to
> handle
> > it appropriately when thrown by libvirt.
>
> Here's the commit that adds the above error message.  In short: Until
> the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY,
the
> copy / mirror block operation is truly not complete.
>
>
>     commit eae59247c59aa02147b2b4a50177e8e877fdb218
>     Author: Peter Krempa <pkrempa@redhat.com>
>     Date:   Wed Jul 15 15:11:02 2015 +0200
>
>         qemu: Update state of block job to READY only if it actually is
> ready
>
>         Few parts of the code looked at the current progress of and
> assumed that
>         a two phase blockjob is in the _READY state as soon as the progress
>         reached 100% (info.cur == info.end). In current versions of qemu
> this
>         assumption is invalid and qemu exposes a new flag 'ready'
in the
>         query-block-jobs output that is set to true if the job is actually
>         finished.
>
>         This patch adds internal data handling for reading the
'ready'
> flag and
>         acting appropriately as long as the flag is present.
>
>         While this still doesn't fix the virsh client problem with two
> phase
>         block jobs and the --pivot option, it at least improves the error
>         message:
>
>         $ virsh blockcommit  --wait --verbose vm vda  --base vda[1]
> --active --pivot
>         Block commit: [100 %]error: failed to pivot job for disk vda
>         error: internal error: unable to execute QEMU command
> 'block-job-complete': The active block job for device
'drive-virtio-disk0'
> cannot be completed
>
>         to
>
>         $ virsh blockcommit  --wait --verbose VM vda  --base vda[1]
> --active --pivot
>         Block commit: [100 %]error: failed to pivot job for disk vda
>         error: block copy still active: disk 'vda' not ready for
pivot yet
>
>
> [...]
>
> --
> /kashyap
>