Marc-Aurèle Brothier - Exoscale
2016-Jun-01 09:59 UTC
[libvirt-users] Migration problem - takes 5 minutes to start moving the memory
Hi, I'm facing a strange issue while doing a migration from an hypervisor to another one. The migration takes for ever to start moving the memory. The VM had no workload what so ever, just a basic ubuntu image. The versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3 Command to launche the migration: virsh migrate --verbose --live --abort-on-error --tunnelled --p2p --auto-converge --copy-storage-inc --xml vm-6160.xml 6160 qemu+tls://<destination_hypervisor>/system Here is the log output, look at the time elapsed: root at virt-hv009:~# virsh domjobinfo 6160 Job type: Unbounded Time elapsed: 27518 ms Data processed: 21.506 GiB Data remaining: 29.003 GiB Data total: 50.509 GiB Memory processed: 0.000 B Memory remaining: 520.820 MiB Memory total: 520.820 MiB File processed: 21.506 GiB File remaining: 28.494 GiB File total: 50.000 GiB Constant pages: 0 Normal pages: 0 Normal data: 0.000 B Expected downtime: 300 ms Setup time: 6 ms root at virt-hv009:~# virsh domjobinfo 6160 Job type: Unbounded Time elapsed: 32331 ms Data processed: 25.475 GiB Data remaining: 25.034 GiB Data total: 50.509 GiB Memory processed: 0.000 B Memory remaining: 520.820 MiB Memory total: 520.820 MiB File processed: 25.475 GiB File remaining: 24.525 GiB File total: 50.000 GiB Constant pages: 0 Normal pages: 0 Normal data: 0.000 B Expected downtime: 300 ms Setup time: 6 ms root at virt-hv009:~# virsh domjobinfo 6160 Job type: Unbounded Time elapsed: 49543 ms Data processed: 50.000 GiB Data remaining: 520.820 MiB Data total: 50.509 GiB Memory processed: 0.000 B Memory remaining: 520.820 MiB Memory total: 520.820 MiB File processed: 50.000 GiB File remaining: 0.000 B File total: 50.000 GiB Constant pages: 0 Normal pages: 0 Normal data: 0.000 B Expected downtime: 300 ms Setup time: 6 ms ^^^^ Here after 49 sec, the disk has been process, but the memory hasn't started I skip the logs, but below is the output 5minutes & 56 seconds later, still nothing: root at virt-hv009:~# virsh domjobinfo 6160 Job type: Unbounded Time elapsed: 356919 ms Data processed: 50.000 GiB Data remaining: 520.820 MiB Data total: 50.509 GiB Memory processed: 0.000 B Memory remaining: 520.820 MiB Memory total: 520.820 MiB File processed: 50.000 GiB File remaining: 0.000 B File total: 50.000 GiB Constant pages: 0 Normal pages: 0 Normal data: 0.000 B Expected downtime: 300 ms Setup time: 6 ms Just after it started to move the memory, but 6 minutes later! root at virt-hv009:~# virsh domjobinfo 6160 Job type: Unbounded Time elapsed: 360092 ms Data processed: 50.052 GiB Data remaining: 453.895 MiB Data total: 50.509 GiB Memory processed: 53.224 MiB Memory remaining: 453.895 MiB Memory total: 520.820 MiB Memory bandwidth: 98.172 MiB/s File processed: 50.000 GiB File remaining: 0.000 B File total: 50.000 GiB Constant pages: 3541 Normal pages: 13591 Normal data: 53.090 MiB Expected downtime: 300 ms Setup time: 6 ms root at virt-hv009:~# virsh domjobinfo 6160 error: failed to get domain '6160' error: Domain not found: no domain with matching name '6160' Here the migration is done. But what made it wait more than 5 minutes to start moving the memory? Does anyone knows? Or have any idea what could produce this latency. - Marco -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20160601/fcc2ad4d/attachment.htm>
Peter Krempa
2016-Jun-01 13:55 UTC
[libvirt-users] Migration problem - takes 5 minutes to start moving the memory
On Wed, Jun 01, 2016 at 11:59:29 +0200, Marc-Aur?le Brothier - Exoscale wrote:> Hi, > > I'm facing a strange issue while doing a migration from an hypervisor to another one. The migration takes for ever to start moving the memory. > The VM had no workload what so ever, just a basic ubuntu image. The versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3 > > Command to launche the migration: > virsh migrate --verbose --live --abort-on-error --tunnelled --p2p --auto-converge --copy-storage-inc --xml vm-6160.xml 6160 qemu+tls://<destination_hypervisor>/system >You are copying storage too. It takes 5 minutes to copy the storage. The memory migration starts after the storage migration converges.
Daniel P. Berrange
2016-Jun-01 13:59 UTC
Re: [libvirt-users] Migration problem - takes 5 minutes to start moving the memory
On Wed, Jun 01, 2016 at 03:55:37PM +0200, Peter Krempa wrote:> On Wed, Jun 01, 2016 at 11:59:29 +0200, Marc-Aurèle Brothier - Exoscale wrote: > > Hi, > > > > I'm facing a strange issue while doing a migration from an hypervisor to another one. The migration takes for ever to start moving the memory. > > The VM had no workload what so ever, just a basic ubuntu image. The versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3 > > > > Command to launche the migration: > > virsh migrate --verbose --live --abort-on-error --tunnelled --p2p --auto-converge --copy-storage-inc --xml vm-6160.xml 6160 qemu+tls://<destination_hypervisor>/system > > > > You are copying storage too. It takes 5 minutes to copy the storage. The > memory migration starts after the storage migration converges.I don't think that's it - if you look at the logs provided, you can see that the storage was apparently fully copied after 49 seconds. There was then 5 minutes where neither the disk or memory processed numbers increased, before memory copying started. So there's something fishy going on there, whether just bogus stats reporting by qemu or a genuine delay/hang somewhere Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|