Jonathan Stoppani
2011-Aug-01 22:11 UTC
[libvirt-users] Migration hangs on Gentoo with KVM
Hi there, I'm trying to migrate a domain between two Gentoo hosts using KVM as hypervisor, but the migration hangs. I tried both live or offline migration modes without success. Details below. Anyone has an idea of what could be wrong with my setup? The migration is the only thing not working. Host A (initially running the VM) ================================ # uname -a Linux pion 2.6.39-gentoo-r3 #12 SMP Fri Jul 29 23:57:49 MST 2011 x86_64 AMD Opteron(tm) Processor 6174 AuthenticAMD GNU/Linux # kvm --version QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard # libvirtd --version libvirtd (libvirt) 0.9.3 # cat /etc/libvirt/libvirtd.conf http://dpaste.com/hold/583874/ # cat /etc/libvirt/qemu.conf http://dpaste.com/hold/583873/ # /etc/init.d/libvirtd start # cat /var/log/libvirt/libvirtd.log http://pastebin.com/V615fgbK # virsh create descriptions/domain.xml # cat descriptions/domain.xml http://dpaste.com/583880/ # cat /var/log/libvirt/libvirtd.log http://pastebin.com/0PSDAZhN (startup log removed) # cat /var/log/libvirt/qemu/nd-50f05b3-0.log http://dpaste.com/583877/ # virsh migrate nd-50f05b3-0 qemu+ssh://host-b/system Password: --> Never exits # cat /var/log/libvirt/libvirtd.log http://pastebin.com/EqHBHx8R (startup and domain creation log removed) # virsh list Id Name State ---------------------------------- 1 nd-50f05b3-0 paused Host B (destination host) ======================== # uname -a Linux lenny 2.6.39-gentoo-r3 #6 SMP Fri Jul 29 18:58:08 MST 2011 x86_64 Intel(R) Xeon(R) CPU 3040 @ 1.86GHz GenuineIntel GNU/Linux # kvm --version QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard # libvirtd --version libvirtd (libvirt) 0.9.3 # cat /etc/libvirt/libvirtd.conf http://dpaste.com/hold/583874/ (same as host A) # cat /etc/libvirt/ http://dpaste.com/hold/583873/ (same as host A) # /etc/init.d/libvirtd start # cat /var/log/libvirt/libvirtd.log http://pastebin.com/WnyqubU6 --> Start migration # cat /var/log/libvirt/libvirtd.log http://pastebin.com/E45JtvGd (startup log removed) # virsh list Id Name State ---------------------------------- 1 nd-50f05b3-0 paused Additional notes =============== * The process seams to deadlock somewhere, CPU usage is near to 0% * The disk images are available on both hosts at the same location (I'm using a COW2 image based on another COW2 image, both are available) * The shared FS is mounted through fuse-sshfs * I just noticed that my clocks are skewed. I will rerun the whole test with a correctly setup ntp server, but I had the same problem last week with perfectly synchronized clocks, so I doubt this will make a difference. Thanks very much, Jonathan
On 08/01/2011 04:11 PM, Jonathan Stoppani wrote:> Hi there, > > I'm trying to migrate a domain between two Gentoo hosts using KVM as hypervisor, but the migration hangs. > I tried both live or offline migration modes without success.Does your 'nc' command have a -q option? If so, then this is probably https://bugzilla.redhat.com/show_bug.cgi?id=726174 and there is a proposed patch for this: https://www.redhat.com/archives/libvir-list/2011-July/msg02041.html although that patch was deemed too late to make it into the upcoming 0.9.4 release. -- Eric Blake eblake at redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
Stefan G. Weichinger
2011-Aug-01 22:26 UTC
[libvirt-users] Migration hangs on Gentoo with KVM
Am 2011-08-02 00:11, schrieb Jonathan Stoppani:> * The process seams to deadlock somewhere, CPU usage is near to 0% * > The disk images are available on both hosts at the same location (I'm > using a COW2 image based on another COW2 image, both are available) * > The shared FS is mounted through fuse-sshfs * I just noticed that my > clocks are skewed. I will rerun the whole test with a correctly setup > ntp server, but I had the same problem last week with perfectly > synchronized clocks, so I doubt this will make a difference.ssh-keys exported/imported vice versa? make sure both hosts are able to ssh to each other using pubkeys. that was my showstopper lately. Stefan
[re-adding the list] On 08/01/2011 04:22 PM, Jonathan Stoppani wrote:> > On Aug 1, 2011, at 16:20 , Eric Blake wrote: > >> On 08/01/2011 04:11 PM, Jonathan Stoppani wrote: >>> Hi there, >>> >>> I'm trying to migrate a domain between two Gentoo hosts using KVM as hypervisor, but the migration hangs. >>> I tried both live or offline migration modes without success. >> >> Does your 'nc' command have a -q option? If so, then this is probably >> >> https://bugzilla.redhat.com/show_bug.cgi?id=726174 >>> > Thanks for the prompt answer Eric! Yes, nc has a q option: > > -q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and remote]Glad to hear that we found root cause to your problems, then.> > The bug specifically refers to ssh, does that mean that it should work over tcp?The problem is that libvirt is trying to start a remote nc session over ssh; but looking at http://libvirt.org/remote.html, it looks like ssh is the only protocol using nc in that manner (so yes, you can probably avoid the issue by using tcp or tls). Meanwhile, I think you can work around it without patching libvirt, by using this as your remote URI: qemu+ssh://user at remotehost/system?netcat=/path/to/nc-wrapper where nc-wrapper is an executable script installed on remotehost, looking like: #!/bin/sh exec /path/to/real/nc -q0 "$@" -- Eric Blake eblake at redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org