libvirt users - Apr 2015 - P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh

Migration without --p2p works just fine, ie. the below works:


    $ virsh migrate --verbose --copy-storage-all \
        --live cvm1 qemu+ssh://kashyapc@devstack3/system
    Migration: [100 %]

    Result:
        - On the source host, the guest is shut off
        - On the destination host, the guest is live migratied successfully


Migration with "--p2p" fails, a simple test below:

First, I should note, I didn't modify any settings in
/etc/libvirt/libvirtd.conf on both source and destination hosts, except for
libvirt logging filters.

(0) On source and destination hosts, SSH keys are setup so that passwordless
    auth works:

    $ ssh-keygen -t 
    $ eval `ssh-agent`
    $ ssh-add .ssh/id_rsa
    $ ssh-copy-id root@devstack3


(1) Check if the connection to the remote host works w/o a prompt for
    user credentials (the below works as user and root):

    $ virsh -c qemu+ssh://kashyapc@devstack3/system
    Welcome to virsh, the virtualization interactive terminal.
    
    Type:  'help' for help with commands
           'quit' to quit
    
    virsh # 


(2) Perform peer to peer live migration (as root):

    $ virsh migrate --verbose --p2p --copy-storage-all \
        --live cvm1 qemu+ssh://kashyapc@devstack3/system
  
    error: operation failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied,
please try again.
    Permission denied, please try again.
    Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).:
Connection reset by peer

>From libvirtd debug log:
[. . .]
2015-04-03 06:04:16.221+0000: 31009: debug : virCommandRunAsync:2408 : About to
run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ssh -l
kashyapc devstack3 sh -c ''\''if '\
''nc'\'' -q 2>&1 | grep "requires an
argument" >/dev/null 2>&1; then ARG=-q0;else
ARG=;fi;'\''nc'\'' $ARG -U
/var/run/libvirt/libvirt-sock'\'''
2015-04-03 06:04:16.223+0000: 31009: debug : virCommandRunAsync:2411 : Command
result 0, with PID 11204
2015-04-03 06:04:16.300+0000: 31009: error : virNetSocketReadWire:1564 : Cannot
recv data: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection
reset by peer
2015-04-03 06:04:16.300+0000: 31009: debug : do_open:1194 : driver 6 remote
returned ERROR
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjExitRemote:1695 :
Exited remote (vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.300+0000: 31009: error : doPeer2PeerMigrate:4711 : operation
failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied,
please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection
reset by peer
2015-04-03 06:04:16.300+0000: 31009: debug :
qemuMigrationRestoreDomainState:1429 : driver=0x7f728c160980, vm=0x7f727c005f80,
pre-mig-state=1, state=1
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjEndAsyncJob:1497 :
Stopping async job: migration out (vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:167 : aborting
child process 11204
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:175 : trying
SIGTERM to child process 11204
[. . .]


What else am I missing?


-- 
/kashyap

On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy
wrote:
> Migration without --p2p works just fine, ie. the below works:
> 
> 
>     $ virsh migrate --verbose --copy-storage-all \
>         --live cvm1 qemu+ssh://kashyapc@devstack3/system
>     Migration: [100 %]
> 
[. . .]
> 
> (2) Perform peer to peer live migration (as root):
> 
>     $ virsh migrate --verbose --p2p --copy-storage-all \
>         --live cvm1 qemu+ssh://kashyapc@devstack3/system
>   
>     error: operation failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied,
please try again.
>     Permission denied, please try again.
>     Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).:
Connection reset by peer
> 
Just to note, both the tests were performed with the below versions on
source and destination hosts:

    kernel-4.0.0-0.rc5.git4.1.fc22.x86_64
    libvirt-daemon-kvm-1.2.13-2.fc22.x86_64
    qemu-system-x86-2.3.0-0.2.rc1.fc22.x86_64

[. . .]

-- 
/kashyap

On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy
wrote:
> On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
> > Migration without --p2p works just fine, ie. the below works:
> > 
> > 
> >     $ virsh migrate --verbose --copy-storage-all \
> >         --live cvm1 qemu+ssh://kashyapc@devstack3/system
> >     Migration: [100 %]
> > 
> 
> [. . .]
> 
> > 
> > (2) Perform peer to peer live migration (as root):
> > 
> >     $ virsh migrate --verbose --p2p --copy-storage-all \
> >         --live cvm1 qemu+ssh://kashyapc@devstack3/system
Hmm, reading the flow for peer2peer here: 

    https://libvirt.org/migration.html#flowpeer2peer

which says:

    "Note that the source libvirtd uses its own credentials (typically
root)
    to connect to the destination, rather than the credentials used by the
    client to connect to the source; if these differ, it is common to run
    into a situation where a client can connect to the destination directly
    but the source cannot make the connection to set up the peer-to-peer
    migration."

So, somehow libvirtd daemon on source is not able to pick the
credentials. I'll try qemu+tcp with the below configs in libvirtd.conf
(note: the auth_tcp = "none" because, I'm doing the test in a
trusted
network).

    listen_tls = 0
    listen_tcp = 1
    auth_tcp = "none"


-- 
/kashyap