libvirt users - Jan 2012 - Libvirt migration issues (0.9.4 and 0.9.9)

Dear all,

we're having two different problems with migrations in libvirt, running as
root user on host machines with CentOS release 5.5 (Final), kernel: Linux
2.6.32.24 #3 SMP Fri Oct 29 16:22:02 BST 2010 x86_64 x86_64 x86_64 GNU/Linux

First case:

virsh version
Compiled against library: libvir 0.9.4
Using library: libvir 0.9.4
Using API: QEMU 0.9.4
Running hypervisor: QEMU 1.0.50

Migrations work well for a basic VM, but if we attach a disk to the usb bus
migration is no longer possible and fails with the error message: "error:
operation failed: migration job: is not active". This is regardless of if
the device is mounted inside the VM or not (debian). Please find more
information attached.

If we attach the same (.iso based) disk to the scsi bus instead, migrations work
as normal.

----

To mitigate this problem, we tried upgrading to a more recent libvirt version:

Compiled against library: libvir 0.9.9
Using library: libvir 0.9.9
Using API: QEMU 0.9.9
Running hypervisor: QEMU 1.0.50

When trying to migrate a normal (debian) instance from one host to another using
the same domain as in the previous successful case without any devices attached,
migration fails with the error message "error: Unable to copy socket file
handle: Invalid argument". The libvirt.log only has a similar single-line
of information:  2012-01-30 15:44:46.772+0000: 7546: error :
virNetSocketDupFD:787 : Unable to copy socket file handle: Invalid argument.

The network configuration used here is the same as we successfully used in the
0.9.4 test case, using static ip's.

Thankful for assistance, not really sure what to try next. :)

Regards,
Daniel Espling


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: libvirt_0.9.4.txt
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20120130/6d031d01/attachment.txt>

Hi again,

I spent some time trying to debug this:

added some printouts and noticed that the virNetSocketDupFD() function is called
with cloexec = True, hence triggering the call:  fd = fcntl(sock->fd,
F_DUPFD_CLOEXEC);

However, running on CentOS 5.5 our glibc version is glibc-2.5-49.el5_5.5, and it
seems the F_DUPFD_CLOEXEC flag was not added until glibc2.7 [1]. I then tried to
replace that code with:

if (cloexec) {
        fd = fcntl(sock->fd, F_DUPFD);
        if (fd >= 0)
            fcntl(fd, F_SETFD, FD_CLOEXEC);
    }

but the results are the same; Unable to copy socket file handle: Invalid
argument

I also added some more printouts to find more about the fd:
2012-01-31 11:20:24.093+0000: 10445: error : virNetSocketDupFD:790 :
sock->fd: 15, cloexec: 1: Invalid argument

and can at least see that fd 15 is the one complaining. Looking at lsof for the
KVM process:

kvm        7456      root   13u     0000                0,8           0       
852 anon_inode
kvm        7456      root   14u     IPv4             167375                   
TCP localhost.localdomain:5900 (LISTEN)
kvm        7456      root   15r     FIFO                0,7                
167376 pipe
kvm        7456      root   16w     FIFO                0,7                
167376 pipe
kvm        7456      root   17u      CHR             10,200                  
2986 /dev/net/tun

Seems like it fails duplicating a pipe leading back to the same process?

Regards,

Daniel

1)
http://stackoverflow.com/questions/1643304/how-to-set-close-on-exec-by-default


On Jan 30, 2012, at 3:14 PM, Daniel Espling wrote:
> Dear all,
> 
> we're having two different problems with migrations in libvirt, running
as root user on host machines with CentOS release 5.5 (Final), kernel: Linux
2.6.32.24 #3 SMP Fri Oct 29 16:22:02 BST 2010 x86_64 x86_64 x86_64 GNU/Linux
> 
> First case:
> 
> virsh version
> Compiled against library: libvir 0.9.4
> Using library: libvir 0.9.4
> Using API: QEMU 0.9.4
> Running hypervisor: QEMU 1.0.50
> 
> Migrations work well for a basic VM, but if we attach a disk to the usb bus
migration is no longer possible and fails with the error message: "error:
operation failed: migration job: is not active". This is regardless of if
the device is mounted inside the VM or not (debian). Please find more
information attached.
> 
> If we attach the same (.iso based) disk to the scsi bus instead, migrations
work as normal.
> 
> ----
> 
> To mitigate this problem, we tried upgrading to a more recent libvirt
version:
> 
> Compiled against library: libvir 0.9.9
> Using library: libvir 0.9.9
> Using API: QEMU 0.9.9
> Running hypervisor: QEMU 1.0.50
> 
> When trying to migrate a normal (debian) instance from one host to another
using the same domain as in the previous successful case without any devices
attached, migration fails with the error message "error: Unable to copy
socket file handle: Invalid argument". The libvirt.log only has a similar
single-line of information:  2012-01-30 15:44:46.772+0000: 7546: error :
virNetSocketDupFD:787 : Unable to copy socket file handle: Invalid argument.
> 
> The network configuration used here is the same as we successfully used in
the 0.9.4 test case, using static ip's.
> 
> Thankful for assistance, not really sure what to try next. :)
> 
> Regards,
> Daniel Espling
> 
> 
> <libvirt_0.9.4.txt>_______________________________________________
> libvirt-users mailing list
> libvirt-users at redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users