libvirt users - Aug 2018 - live migration via unix socket

Hey,

Over in KubeVirt we're investigating a use case where we'd like to
perform
a live migration within a network namespace that does not provide libvirtd
with network access. In this scenario we would like to perform a live
migration by proxying the migration through a unix socket to a process in
another network namespace that does have network access.  That external
process would live on every node in the cluster and know how to correctly
route connections between libvirtds.

virsh example of an attempted migration via unix socket.

virsh migrate --copy-storage-all --p2p --live --xml domain.xml my-vm
qemu+unix:///system?socket=destination-host-proxy-sock

In this example, the src libvirtd is able to establish a connection to the
destination libvirtd via the unix socket proxy. However, the migration-uri
appears to require either tcp or rdma network connection.  If I force the
migration-uri to be a unix socket, I receive an error [1] indicating that
qemu+unix is not a valid transport.

Technically with qemu+kvm I believe what we're attempting should be
possible (even though it is inefficient). Please correct me if I'm wrong.

Is there a way to achieve this migration via unix socket functionality this
using Libvirt? Also, is there a reason why the migration uri is limited to
tcp/rdma

Thanks!
- David

[1]
https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_migration.c#L2798

On Tue, Aug 28, 2018 at 05:07:18PM -0400, David Vossel
wrote:
> Hey,
> 
> Over in KubeVirt we're investigating a use case where we'd like to
perform
> a live migration within a network namespace that does not provide libvirtd
> with network access. In this scenario we would like to perform a live
> migration by proxying the migration through a unix socket to a process in
> another network namespace that does have network access.  That external
> process would live on every node in the cluster and know how to correctly
> route connections between libvirtds.
> 
> virsh example of an attempted migration via unix socket.
> 
> virsh migrate --copy-storage-all --p2p --live --xml domain.xml my-vm
> qemu+unix:///system?socket=destination-host-proxy-sock
> 
> In this example, the src libvirtd is able to establish a connection to the
> destination libvirtd via the unix socket proxy. However, the migration-uri
> appears to require either tcp or rdma network connection.  If I force the
> migration-uri to be a unix socket, I receive an error [1] indicating that
> qemu+unix is not a valid transport.
qemu+unix is a syntax for libvirt's URI format.  The URI scheme for
migration is not the same, so you can't simply plug in qemu+unix here.
> 
> Technically with qemu+kvm I believe what we're attempting should be
> possible (even though it is inefficient). Please correct me if I'm
wrong.
> 
> Is there a way to achieve this migration via unix socket functionality this
> using Libvirt? Also, is there a reason why the migration uri is limited to
> tcp/rdma
Internally libvirt does exactly this when using its TUNNELLED live migration
mode. In this QEMU is passed an anonymous UNIX socket and the data is all
copied over the libvirtd <-> libvirtd connection and then copied again
back
to QEMU on another UNIX socket. This was done because QEMU has long had no
ability to encrypt live migration, so tunnelling over libvirtd's own TLS
secured connection was only secure mechanism.

We've done work in QEMU to natively support TLS now so that we can get rid
of this tunnelling, as this architecture decreased performance and consumed
precious CPU memory bandwidth, which is particularly bad when libvirtd and
QEMU were on different NUMA nodes. It is already a challenge to get live
migration to successfully complete even with a direct network connection.
Although QEMU can do it at the low level, we've never exposed anything
other than direct network transports at the API level. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, Aug 29, 2018 at 4:55 AM, Daniel P. Berrangé <berrange@redhat.com>
wrote:
> On Tue, Aug 28, 2018 at 05:07:18PM -0400, David Vossel wrote:
> > Hey,
> >
> > Over in KubeVirt we're investigating a use case where we'd
like to
> perform
> > a live migration within a network namespace that does not provide
> libvirtd
> > with network access. In this scenario we would like to perform a live
> > migration by proxying the migration through a unix socket to a process
in
> > another network namespace that does have network access.  That
external
> > process would live on every node in the cluster and know how to
correctly
> > route connections between libvirtds.
> >
> > virsh example of an attempted migration via unix socket.
> >
> > virsh migrate --copy-storage-all --p2p --live --xml domain.xml my-vm
> > qemu+unix:///system?socket=destination-host-proxy-sock
> >
> > In this example, the src libvirtd is able to establish a connection to
> the
> > destination libvirtd via the unix socket proxy. However, the
> migration-uri
> > appears to require either tcp or rdma network connection.  If I force
the
> > migration-uri to be a unix socket, I receive an error [1] indicating
that
> > qemu+unix is not a valid transport.
>
> qemu+unix is a syntax for libvirt's URI format.  The URI scheme for
> migration is not the same, so you can't simply plug in qemu+unix here.
>
> >
> > Technically with qemu+kvm I believe what we're attempting should
be
> > possible (even though it is inefficient). Please correct me if I'm
wrong.
> >
> > Is there a way to achieve this migration via unix socket functionality
> this
> > using Libvirt? Also, is there a reason why the migration uri is
limited
> to
> > tcp/rdma
>
> Internally libvirt does exactly this when using its TUNNELLED live
> migration
> mode. In this QEMU is passed an anonymous UNIX socket and the data is all
> copied over the libvirtd <-> libvirtd connection and then copied
again back
>
Sorry for the delayed response here, I've only just picked this task back
up again recently.

With the TUNNELLED and PEER2PEER migration flags set, Libvirt won't allow
the libvirtd <-> libvirtd connection over a unix socket.

Libvirt returns this error "Attempt to migrate guest to the same
host".
The virDomainMigrateCheckNotLocal() function ensures that a peer2peer
migration won't occur when the destination is a unix socket.

Is there anyway around this? We'd like to tunnel the destination connection
through a unix socket. The other side of the unix socket is a network proxy
in a different network namespace which properly performs the remote
connection.

> to QEMU on another UNIX socket. This was done because QEMU has long had no
> ability to encrypt live migration, so tunnelling over libvirtd's own
TLS
> secured connection was only secure mechanism.
> We've done work in QEMU to natively support TLS now so that we can get
rid
> of this tunnelling, as this architecture decreased performance and consumed
> precious CPU memory bandwidth, which is particularly bad when libvirtd and
> QEMU were on different NUMA nodes. It is already a challenge to get live
> migration to successfully complete even with a direct network connection.
> Although QEMU can do it at the low level, we've never exposed anything
> other than direct network transports at the API level.
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/
> dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/
> dberrange :|
>