libvirt users - Jun 2016 - Migration problem - takes 5 minutes to start moving the memory

Hi,

I'm facing a strange issue while doing a migration from an hypervisor to
another one. The migration takes for ever to start moving the memory.
The VM had no workload what so ever, just a basic ubuntu image. The versions on
the hypervisors are: libvirt 1.2.21, qemu 1.2.3

Command to launche the migration:
virsh migrate --verbose --live --abort-on-error --tunnelled --p2p
--auto-converge --copy-storage-inc --xml vm-6160.xml 6160 
qemu+tls://<destination_hypervisor>/system

Here is the log output, look at the time elapsed:

root at virt-hv009:~# virsh domjobinfo 6160
Job type:         Unbounded   
Time elapsed:     27518        ms
Data processed:   21.506 GiB
Data remaining:   29.003 GiB
Data total:       50.509 GiB
Memory processed: 0.000 B
Memory remaining: 520.820 MiB
Memory total:     520.820 MiB
File processed:   21.506 GiB
File remaining:   28.494 GiB
File total:       50.000 GiB
Constant pages:   0           
Normal pages:     0           
Normal data:      0.000 B
Expected downtime: 300          ms
Setup time:       6            ms

root at virt-hv009:~# virsh domjobinfo 6160
Job type:         Unbounded   
Time elapsed:     32331        ms
Data processed:   25.475 GiB
Data remaining:   25.034 GiB
Data total:       50.509 GiB
Memory processed: 0.000 B
Memory remaining: 520.820 MiB
Memory total:     520.820 MiB
File processed:   25.475 GiB
File remaining:   24.525 GiB
File total:       50.000 GiB
Constant pages:   0           
Normal pages:     0           
Normal data:      0.000 B
Expected downtime: 300          ms
Setup time:       6            ms

root at virt-hv009:~# virsh domjobinfo 6160
Job type:         Unbounded   
Time elapsed:     49543        ms
Data processed:   50.000 GiB
Data remaining:   520.820 MiB
Data total:       50.509 GiB
Memory processed: 0.000 B
Memory remaining: 520.820 MiB
Memory total:     520.820 MiB
File processed:   50.000 GiB
File remaining:   0.000 B
File total:       50.000 GiB
Constant pages:   0           
Normal pages:     0           
Normal data:      0.000 B
Expected downtime: 300          ms
Setup time:       6            ms


^^^^ Here after 49 sec, the disk has been process, but the memory hasn't
started

I skip the logs, but below is the output 5minutes & 56 seconds later, still
nothing:

root at virt-hv009:~# virsh domjobinfo 6160
Job type:         Unbounded   
Time elapsed:     356919       ms
Data processed:   50.000 GiB
Data remaining:   520.820 MiB
Data total:       50.509 GiB
Memory processed: 0.000 B
Memory remaining: 520.820 MiB
Memory total:     520.820 MiB
File processed:   50.000 GiB
File remaining:   0.000 B
File total:       50.000 GiB
Constant pages:   0           
Normal pages:     0           
Normal data:      0.000 B
Expected downtime: 300          ms
Setup time:       6            ms

Just after it started to move the memory, but 6 minutes later!

root at virt-hv009:~# virsh domjobinfo 6160
Job type:         Unbounded   
Time elapsed:     360092       ms
Data processed:   50.052 GiB
Data remaining:   453.895 MiB
Data total:       50.509 GiB
Memory processed: 53.224 MiB
Memory remaining: 453.895 MiB
Memory total:     520.820 MiB
Memory bandwidth: 98.172 MiB/s
File processed:   50.000 GiB
File remaining:   0.000 B
File total:       50.000 GiB
Constant pages:   3541        
Normal pages:     13591       
Normal data:      53.090 MiB
Expected downtime: 300          ms
Setup time:       6            ms

root at virt-hv009:~# virsh domjobinfo 6160
error: failed to get domain '6160'
error: Domain not found: no domain with matching name '6160'

Here the migration is done. But what made it wait more than 5 minutes to start
moving the memory? Does anyone knows? Or have any idea what could produce this
latency.

- Marco
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20160601/fcc2ad4d/attachment.htm>

On Wed, Jun 01, 2016 at 11:59:29 +0200, Marc-Aur?le Brothier - Exoscale
wrote:
> Hi,
> 
> I'm facing a strange issue while doing a migration from an hypervisor
to another one. The migration takes for ever to start moving the memory.
> The VM had no workload what so ever, just a basic ubuntu image. The
versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3
> 
> Command to launche the migration:
> virsh migrate --verbose --live --abort-on-error --tunnelled --p2p
--auto-converge --copy-storage-inc --xml vm-6160.xml 6160 
qemu+tls://<destination_hypervisor>/system
> 
You are copying storage too. It takes 5 minutes to copy the storage. The
memory migration starts after the storage migration converges.

On Wed, Jun 01, 2016 at 03:55:37PM +0200, Peter Krempa
wrote:
> On Wed, Jun 01, 2016 at 11:59:29 +0200, Marc-Aurèle Brothier - Exoscale
wrote:
> > Hi,
> > 
> > I'm facing a strange issue while doing a migration from an
hypervisor to another one. The migration takes for ever to start moving the
memory.
> > The VM had no workload what so ever, just a basic ubuntu image. The
versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3
> > 
> > Command to launche the migration:
> > virsh migrate --verbose --live --abort-on-error --tunnelled --p2p
--auto-converge --copy-storage-inc --xml vm-6160.xml 6160 
qemu+tls://<destination_hypervisor>/system
> > 
> 
> You are copying storage too. It takes 5 minutes to copy the storage. The
> memory migration starts after the storage migration converges.
I don't think that's it - if you look at the logs provided, you can see
that the storage was apparently fully copied after 49 seconds. There was
then 5 minutes where neither the disk or memory processed numbers
increased, before memory copying started. So there's something fishy
going on there, whether just bogus stats reporting by qemu or a genuine
delay/hang somewhere

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|