libvirt users - Apr 2022 - race condition? virsh migrate --copy-storage-all

Hi list,

I'm trying to migrate a few qemu virtual machines between two 1G 
ethernet connected hosts, with local storage only. I got endless "error: 
operation failed: migration of disk vda failed: Input/output error" 
errors and thought: something wrong with settings.

However, then, suddenly: I succeeded without changing anything. And, hey:
  while ! time virsh migrate --live --persistent --undefinesource 
--copy-storage-all ubuntu20.04 qemu+ssh://duikboot/system; do a=$(( $a + 
1 )); echo $a; done

... retried 8 times, but then: success. This smells like a race 
condition, doesn't it? A bit weird is the fact that the migration seems 
to succeed every time while copying from revolving disks to SSD; but the 
other way around has this Input/output error.

There are some messages in /var/log/syslog, but not at the time of the 
failure, and no disk errors. These disks are LVM2 volumes and they live 
on raid arrays - and/so there is not a real, as in physical, I/O-error.

Source system has SSD's, target system has regular disks.

1) is this the right mailing list? I'm not 100% sure.
2) how can I research this further? Spending hours on a "while / then"
loop to try and retry live migration looks like a dull job for my poor 
computers ;-)

Best regards,

Valentijn

On Fri, Apr 15, 2022 at 16:58:08 +0200, Valentijn Sessink
wrote:
> Hi list,
> 
> I'm trying to migrate a few qemu virtual machines between two 1G
ethernet
> connected hosts, with local storage only. I got endless "error:
operation
> failed: migration of disk vda failed: Input/output error" errors and
> thought: something wrong with settings.
> 
> However, then, suddenly: I succeeded without changing anything. And, hey:
>  while ! time virsh migrate --live --persistent --undefinesource
> --copy-storage-all ubuntu20.04 qemu+ssh://duikboot/system; do a=$(( $a + 1
> )); echo $a; done
> 
> ... retried 8 times, but then: success. This smells like a race condition,
> doesn't it? A bit weird is the fact that the migration seems to succeed
> every time while copying from revolving disks to SSD; but the other way
> around has this Input/output error.
> 
> There are some messages in /var/log/syslog, but not at the time of the
> failure, and no disk errors. These disks are LVM2 volumes and they live on
> raid arrays - and/so there is not a real, as in physical, I/O-error.
> 
> Source system has SSD's, target system has regular disks.
> 
> 1) is this the right mailing list? I'm not 100% sure.
> 2) how can I research this further? Spending hours on a "while /
then" loop
> to try and retry live migration looks like a dull job for my poor computers
> ;-)
It would be helpful if you provide the VM XML file to see how your disks
are configured and the debug log file when the bug reproduces:

https://www.libvirt.org/kbase/debuglogs.html#less-verbose-logging-for-qemu-vms

Without that my only hunch would be that you ran out of disk space on
the destination which caused the I/O error.