similar to: [nbdkit PATCH] nbd: Fix gcc warning and off-by-one in socket name length

Accepting only Unix sockets can be a bit limiting; let's be more flexible. Eric Blake (2): nbd: Refactor Unix socket connection nbd: Support TCP socket plugins/nbd/nbdkit-nbd-plugin.pod | 36 ++++-- plugins/nbd/nbd.c | 175 ++++++++++++++++++++++-------- TODO | 3 - 3 files changed, 161 insertions(+), 53 deletions(-) -- 2.20.1

I've documented a desire to do this for a while, time to actually follow through and support connecting as a client to a TCP server. Note that it is desirable to support the plugin connecting to an encrypted server over TCP, then exposing the raw data over a local Unix socket; that aspect requires yet more work, left for another day. But even allowing an old-style client to connect to an

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket with no encryption, and the transactions are serialized

I'm still working on the interleaving (and Rich reminded me on IRC that we still don't have THREAD_MODEL_PARALLEL working anywhere yet, anyways). Since nbdkit doesn't really have a parallel plugin yet, my testing on that front will have to use qemu-nbd as the original server, as well as qemu-io as the driver (qemu-io's aio_read and aio_write commands can be used to trigger

When we first used libnbd (nbdkit 1.14), it was brand new, so we left in the ability to build a limited nbd plugin without the use of libnbd for comparison. But now that libnbd has had more time under its belt, and we have learned that our nbd-standalone code is not getting much testing, it's time to follow through with our plan to finish off the old code, and now build the nbd plugin only

qemu-nbd defaults to permitting only a single NBD client. Of course, you can run qemu-nbd -t to work around it, but other servers may have similar restrictions, at which point nbdkit's nbd plugin can provide the automatic fanout to multiple clients via a single server link. Note that the shared=true parameter makes the previously-added retry=N parameter more useful, as it is much easier to

Copy various Unix socket handling techniques from nbdkit's nbd plugin: Silently truncating a socket name rather than issuing an error message can confuse users. No need to do an explicit memset if the compiler does it for us via an initializer. No need to use strncpy() which does wasted effort on short names, when we can use memcpy() given that we already checked length in order to detect

I got annoyed by qemu-nbd's default of only allowing a single connection; combine that with nbdkit's nbd plugin, and even 'qemu-nbd --list' of nbdkit counts as the single connection and immediately hangs up. If we introduce a shared mode, then 'qemu-nbd --list' can connect as many times as it wants without killing the original qemu-nbd wrapped by nbdkit. But this in turn

With new enough libnbd, we already support vsock by virtue of uri=; however, it's also nice to allow direct exposure of the nbd_connect_vsock() api. Signed-off-by: Eric Blake <eblake@redhat.com> --- As with commit 7ce9feef, there is no easy way to add testsuite coverage for this. plugins/nbd/nbdkit-nbd-plugin.pod | 30 +++++++++----- plugins/nbd/nbd.c | 65

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd:

Error out early if the path to the socket will not fit into sockaddr_un::sun_path, as we will not be able to connect to it. --- src/launch-unix.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/launch-unix.c b/src/launch-unix.c index 740c554..973e14b 100644 --- a/src/launch-unix.c +++ b/src/launch-unix.c @@ -47,6 +47,12 @@ launch_unix (guestfs_h *g, void *datav, const char

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket, and the transactions are serialized rather than

libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing. Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+, add tests to TLS usage which flushed out the need to turn relative pathnames into absolute, doc tweaks Now that the testsuite covers TLS and libnbd has been fixed to provide the things I found lacking when developing v2, I'm leaning towards pushing this on

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros.

In preparation for allowing interleaved response, refactor the nbd forwarder so that writes are still done from the thread handling the original request from the client, but all reads are done by a dedicated reader thread. Control between the two flags is gated by a mutex for storing the transaction information, coupled with a pipe for the reader thread to send the final status back to the

Found by Coverity. --- plugins/nbd/nbd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index b9e72bc..2b5569b 100644 --- a/plugins/nbd/nbd.c +++ b/plugins/nbd/nbd.c @@ -465,6 +465,7 @@ nbd_open (int readonly) h->fd = socket (AF_UNIX, SOCK_STREAM, 0); if (h->fd < 0) { nbdkit_error ("socket: %m"); + free (h); return

There are a few other issues that Coverity found, but I believe all can be ignored ... except one: We don't set umask anywhere inside nbdkit. Coverity complains that this is a problem where we create temporary files, since the result of mkstemp depends implicitly on the umask value. I think we might consider setting umask anyway (eg. to 022) just to make plugin behaviour more predictable.

With this, I am finally able to get the nbd plugin to do out-of-order responses to the client. Once this series goes in, we should be ready for Rich to cut a release. Eric Blake (4): nbd: Split reading into separate thread nbd: Protect writes with mutex nbd: Enable parallel handling tests: Test parallel nbd behavior plugins/nbd/nbd.c | 217

Just tried out ssh in pre13, and still get the EGD problem? :( new-relay:/usr/slocal/src/openssh-1.2pre13> ./ssh -l marc atelier The authenticity of host 'atelier.acadiau.ca' can't be established. Key fingerprint is 1024 ef:36:b5:f8:a3:bb:14:4d:a9:4b:f2:90:9a:bd:bb:00. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added

This conversion should be feature compatible with the standalone nbd code. Note that the use of libnbd makes the binary for this particular plugin fall under an LGPLv2+ license rather than BSD; but the source code in nbd.c remains BSD. A lot of code simply disappears, now that I'm no longer directly utilizing the NBD protocol files but relying on libnbd. Coordination between threads from