Kashyap Chamarthy
2015-Apr-03 08:08 UTC
[libvirt-users] P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
Migration: [100 %]
Result:
- On the source host, the guest is shut off
- On the destination host, the guest is live migratied successfully
Migration with "--p2p" fails, a simple test below:
First, I should note, I didn't modify any settings in
/etc/libvirt/libvirtd.conf on both source and destination hosts, except for
libvirt logging filters.
(0) On source and destination hosts, SSH keys are setup so that passwordless
auth works:
$ ssh-keygen -t
$ eval `ssh-agent`
$ ssh-add .ssh/id_rsa
$ ssh-copy-id root@devstack3
(1) Check if the connection to the remote host works w/o a prompt for
user credentials (the below works as user and root):
$ virsh -c qemu+ssh://kashyapc@devstack3/system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh #
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
error: operation failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied,
please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).:
Connection reset by peer
>From libvirtd debug log:
[. . .]
2015-04-03 06:04:16.221+0000: 31009: debug : virCommandRunAsync:2408 : About to
run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ssh -l
kashyapc devstack3 sh -c ''\''if '\
''nc'\'' -q 2>&1 | grep "requires an
argument" >/dev/null 2>&1; then ARG=-q0;else
ARG=;fi;'\''nc'\'' $ARG -U
/var/run/libvirt/libvirt-sock'\'''
2015-04-03 06:04:16.223+0000: 31009: debug : virCommandRunAsync:2411 : Command
result 0, with PID 11204
2015-04-03 06:04:16.300+0000: 31009: error : virNetSocketReadWire:1564 : Cannot
recv data: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection
reset by peer
2015-04-03 06:04:16.300+0000: 31009: debug : do_open:1194 : driver 6 remote
returned ERROR
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjExitRemote:1695 :
Exited remote (vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.300+0000: 31009: error : doPeer2PeerMigrate:4711 : operation
failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied,
please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection
reset by peer
2015-04-03 06:04:16.300+0000: 31009: debug :
qemuMigrationRestoreDomainState:1429 : driver=0x7f728c160980, vm=0x7f727c005f80,
pre-mig-state=1, state=1
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjEndAsyncJob:1497 :
Stopping async job: migration out (vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:167 : aborting
child process 11204
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:175 : trying
SIGTERM to child process 11204
[. . .]
What else am I missing?
--
/kashyap
Kashyap Chamarthy
2015-Apr-03 08:13 UTC
Re: [libvirt-users] P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:> Migration without --p2p works just fine, ie. the below works: > > > $ virsh migrate --verbose --copy-storage-all \ > --live cvm1 qemu+ssh://kashyapc@devstack3/system > Migration: [100 %] >[. . .]> > (2) Perform peer to peer live migration (as root): > > $ virsh migrate --verbose --p2p --copy-storage-all \ > --live cvm1 qemu+ssh://kashyapc@devstack3/system > > error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try again. > Permission denied, please try again. > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer >Just to note, both the tests were performed with the below versions on source and destination hosts: kernel-4.0.0-0.rc5.git4.1.fc22.x86_64 libvirt-daemon-kvm-1.2.13-2.fc22.x86_64 qemu-system-x86-2.3.0-0.2.rc1.fc22.x86_64 [. . .] -- /kashyap
Kashyap Chamarthy
2015-Apr-03 10:27 UTC
Re: [libvirt-users] P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy wrote:> On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote: > > Migration without --p2p works just fine, ie. the below works: > > > > > > $ virsh migrate --verbose --copy-storage-all \ > > --live cvm1 qemu+ssh://kashyapc@devstack3/system > > Migration: [100 %] > > > > [. . .] > > > > > (2) Perform peer to peer live migration (as root): > > > > $ virsh migrate --verbose --p2p --copy-storage-all \ > > --live cvm1 qemu+ssh://kashyapc@devstack3/systemHmm, reading the flow for peer2peer here: https://libvirt.org/migration.html#flowpeer2peer which says: "Note that the source libvirtd uses its own credentials (typically root) to connect to the destination, rather than the credentials used by the client to connect to the source; if these differ, it is common to run into a situation where a client can connect to the destination directly but the source cannot make the connection to set up the peer-to-peer migration." So, somehow libvirtd daemon on source is not able to pick the credentials. I'll try qemu+tcp with the below configs in libvirtd.conf (note: the auth_tcp = "none" because, I'm doing the test in a trusted network). listen_tls = 0 listen_tcp = 1 auth_tcp = "none" -- /kashyap