Libguestfs - Oct 2019 - Re: [PATCH libnbd 2/2] api: Add support for AF_VSOCK.

On 10/18/19 10:39 AM, Richard W.M. Jones wrote:
> This adds a new API for connecting to AF_VSOCK protocol
> (https://wiki.qemu.org/Features/VirtioVsock).
> 
> For example:
> 
>    nbd_connect_vsock (nbd, 2, 10809);
> 
> There is no test of this feature because it only works between guest
> and host.  You cannot start a server and client on the host and talk
> between them, which is what we'd need to write a sane test.
> ---
> @@ -357,6 +366,23 @@ main (int argc, char *argv[])
>         exit (EXIT_FAILURE);
>       }
>       break;
> +
> +  case MODE_VSOCK:
> +    if (sscanf (argv[optind], "%" SCNu32, &cid) != 1) {
sscanf() can't detect overflow ;(
> +++ b/generator/generator
> +  "connect_vsock", {
> +    default_call with
> +    args = [ UInt32 "cid"; UInt32 "port" ]; ret =
RErr;
> +    permitted_states = [ Created ];
> +    shortdesc = "connect to NBD server over AF_VSOCK protocol";
> +    longdesc = "\
> +Connect (synchronously) over the C<AF_VSOCK> protocol from a
> +virtual machine to an NBD server, usually running on the host.  The
> +C<cid> and C<port> parameters specify the server address. 
Usually
> +C<cid> should be C<2> (to connect to the host), and
C<port> might be
> +C<10809> or another port number assigned to you by the host
> +administrator.  This call returns when the connection has been
made.";
You mentioned that right now, nbdkit has to be server on host, and 
libnbd is client on guest.  But if we can let nbdkit specify a cid, 
doesn't this mean we can run nbdkit as server in guest, and then connect 
libnbd as client on host?  Then add 'nbdkit nbd vsock=...' to let the 
nbdkit pass-through wrapper convert vsock from guest into TCP or Unix 
socket on the host to other host clients that don't know how to do vsock.

> @@ -2793,6 +2836,8 @@ let first_version = [
>     "aio_connect_systemd_socket_activation", (1, 2);
>     "connect_socket", (1, 2);
>     "aio_connect_socket", (1, 2);
> +  "connect_vsock", (1, 2);
> +  "aio_connect_vsock", (1, 2);
>   
As this is Linux-only (and for that matter, depends on kernel vsock 
support), we probably need "supports_vsock" as an additional function.
> +++ b/generator/states-connect.c
> @@ -37,6 +37,10 @@
>   #include <sys/socket.h>
>   #include <sys/un.h>
>   
> +#ifdef HAVE_LINUX_VM_SOCKETS_H
> +#include <linux/vm_sockets.h>
> +#endif
> +
>   /* Disable Nagle's algorithm on the socket, but don't fail. */
>   static void
>   disable_nagle (int sock)
> @@ -118,6 +122,24 @@ STATE_MACHINE {
>     SET_NEXT_STATE (%^CONNECT.START);
>     return 0;
>   
> + CONNECT_VSOCK.START:
> +#ifdef AF_VSOCK
> +  struct sockaddr_vm svm = {
> +    .svm_family = AF_VSOCK,
> +    .svm_cid = h->svm_cid,
> +    .svm_port = h->svm_port,
> +  };
Are there scenarios (mismatch in kernel vs. headers compiled against, 
for instance) where compilation says AF_VSOCK exists but where all 
attempts at vsock fail?  If so, is there anything that we should check 
dynamically, rather than just compile-time presence of AF_VSOCK?  But if 
nothing else, having:

int
libnbd_internal_supports_vsock(struct nbd_handle*h)
{
#ifdef AF_VSOCK
   return 1;
#else
   return 0;
}

is worth having.

Otherwise, the idea is pretty cool!

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On Fri, Oct 18, 2019 at 10:58:43AM -0500, Eric Blake
wrote:
> On 10/18/19 10:39 AM, Richard W.M. Jones wrote:
> >+Connect (synchronously) over the C<AF_VSOCK> protocol from a
> >+virtual machine to an NBD server, usually running on the host.  The
> >+C<cid> and C<port> parameters specify the server address. 
Usually
> >+C<cid> should be C<2> (to connect to the host), and
C<port> might be
> >+C<10809> or another port number assigned to you by the host
> >+administrator.  This call returns when the connection has been
made.";
> 
> You mentioned that right now, nbdkit has to be server on host, and
> libnbd is client on guest.  But if we can let nbdkit specify a cid,
> doesn't this mean we can run nbdkit as server in guest, and then
> connect libnbd as client on host?  Then add 'nbdkit nbd vsock=...'
> to let the nbdkit pass-through wrapper convert vsock from guest into
> TCP or Unix socket on the host to other host clients that don't know
> how to do vsock.
I'm afraid I don't understand this.
> >@@ -2793,6 +2836,8 @@ let first_version = [
> >    "aio_connect_systemd_socket_activation", (1, 2);
> >    "connect_socket", (1, 2);
> >    "aio_connect_socket", (1, 2);
> >+  "connect_vsock", (1, 2);
> >+  "aio_connect_vsock", (1, 2);
> 
> As this is Linux-only (and for that matter, depends on kernel vsock
> support), we probably need "supports_vsock" as an additional
> function.
We could add it, but equally client code can just check AF_VSOCK the
same way that libnbd does, ie with a configure test for the header and
checking if AF_VSOCK is defined.
> >+++ b/generator/states-connect.c
> >@@ -37,6 +37,10 @@
> >  #include <sys/socket.h>
> >  #include <sys/un.h>
> >+#ifdef HAVE_LINUX_VM_SOCKETS_H
> >+#include <linux/vm_sockets.h>
> >+#endif
> >+
> >  /* Disable Nagle's algorithm on the socket, but don't fail.
*/
> >  static void
> >  disable_nagle (int sock)
> >@@ -118,6 +122,24 @@ STATE_MACHINE {
> >    SET_NEXT_STATE (%^CONNECT.START);
> >    return 0;
> >+ CONNECT_VSOCK.START:
> >+#ifdef AF_VSOCK
> >+  struct sockaddr_vm svm = {
> >+    .svm_family = AF_VSOCK,
> >+    .svm_cid = h->svm_cid,
> >+    .svm_port = h->svm_port,
> >+  };
> 
> Are there scenarios (mismatch in kernel vs. headers compiled
> against, for instance) where compilation says AF_VSOCK exists but
> where all attempts at vsock fail?  If so, is there anything that we
> should check dynamically, rather than just compile-time presence of
> AF_VSOCK?
You can actually test this just by running the following command on
the host:

$ ./run strace -f -o /tmp/log nbdsh -c 'h.connect_vsock(2, 10809)'

The log shows how it fails:

3395572 socket(AF_VSOCK, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
...
3395572 connect(3, {sa_family=AF_VSOCK,
sa_data="\0\09*\0\0\2\0\0\0\0\0\0\0"}, 16) = -1 ENODEV (No such
device)

That's unfortunate as it means we would have to connect to a
non-existent socket and determine if it returns ENODEV or ECONNREFUSED
(and if it returns something else - erm?!)  Note that the socket(2)
syscall doesn't fail because creating a server socket is perfectly
valid whether in a host or guest context.

Rich.
> But if nothing else, having:
> int
> libnbd_internal_supports_vsock(struct nbd_handle*h)
> {
> #ifdef AF_VSOCK
>   return 1;
> #else
>   return 0;
> }
> 
> is worth having.
> 
> Otherwise, the idea is pretty cool!
> 
> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3226
> Virtualization:  qemu.org | libvirt.org
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Fri, Oct 18, 2019 at 05:34:17PM +0100, Richard W.M. Jones
wrote:
> On Fri, Oct 18, 2019 at 10:58:43AM -0500, Eric Blake wrote:
> > On 10/18/19 10:39 AM, Richard W.M. Jones wrote:
> > >+Connect (synchronously) over the C<AF_VSOCK> protocol from
a
> > >+virtual machine to an NBD server, usually running on the host. 
The
> > >+C<cid> and C<port> parameters specify the server
address.  Usually
> > >+C<cid> should be C<2> (to connect to the host), and
C<port> might be
> > >+C<10809> or another port number assigned to you by the host
> > >+administrator.  This call returns when the connection has been
made.";
> > 
> > You mentioned that right now, nbdkit has to be server on host, and
> > libnbd is client on guest.  But if we can let nbdkit specify a cid,
> > doesn't this mean we can run nbdkit as server in guest, and then
> > connect libnbd as client on host?  Then add 'nbdkit nbd
vsock=...'
> > to let the nbdkit pass-through wrapper convert vsock from guest into
> > TCP or Unix socket on the host to other host clients that don't
know
> > how to do vsock.
> 
> I'm afraid I don't understand this.
After re-reading it I do understand this and also the need for it.
However I was able to fix it another way, by having nbdkit listen on
the "ANY" address, so now it should be possible to run it from the
guest too.

It would still be useful to add vsock capability to nbdkit-nbd-plugin
for other reasons (NBD clients which don't support vsock but which
need to access a vsock resource).

Note I have added support for vsock URIs to libnbd now too, using the
obvious syntax (nbd+vsock:/// or nbd+vsock://2:10809/)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Since Linux >= 5.6 supports vsock in loopback mode, I was able to add
a regression test for this feature to nbdkit / libnbd, and I can
report that it appears to work fine.

https://github.com/libguestfs/nbdkit/commit/5e4745641bb4676f607fdb3f8750dbf6e9516877

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Mon, Feb 10, 2020 at 12:45:47PM +0000, Richard W.M. Jones
wrote:
> Since Linux >= 5.6 supports vsock in loopback mode, I was able to add
> a regression test for this feature to nbdkit / libnbd, and I can
> report that it appears to work fine.
> 
>
https://github.com/libguestfs/nbdkit/commit/5e4745641bb4676f607fdb3f8750dbf6e9516877
> 
I'm happy that it works and it is useful :-)
Thanks again for the suggestion!

Just a note, Linux >= 5.6 supports a new special cid (VMADDR_CID_LOCAL = 1)
to use the loopback mode in any condition (host, guest, nested guest).

Cheers,
Stefano