similar to: [RFC nbdkit PATCH 0/2] Add 'nbdkit nbd shared=1' mode

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros.

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd:

libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing. Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+, add tests to TLS usage which flushed out the need to turn relative pathnames into absolute, doc tweaks Now that the testsuite covers TLS and libnbd has been fixed to provide the things I found lacking when developing v2, I'm leaning towards pushing this on

Accepting only Unix sockets can be a bit limiting; let's be more flexible. Eric Blake (2): nbd: Refactor Unix socket connection nbd: Support TCP socket plugins/nbd/nbdkit-nbd-plugin.pod | 36 ++++-- plugins/nbd/nbd.c | 175 ++++++++++++++++++++++-------- TODO | 3 - 3 files changed, 161 insertions(+), 53 deletions(-) -- 2.20.1

qemu-nbd defaults to permitting only a single NBD client. Of course, you can run qemu-nbd -t to work around it, but other servers may have similar restrictions, at which point nbdkit's nbd plugin can provide the automatic fanout to multiple clients via a single server link. Note that the shared=true parameter makes the previously-added retry=N parameter more useful, as it is much easier to

I'm still working on the interleaving (and Rich reminded me on IRC that we still don't have THREAD_MODEL_PARALLEL working anywhere yet, anyways). Since nbdkit doesn't really have a parallel plugin yet, my testing on that front will have to use qemu-nbd as the original server, as well as qemu-io as the driver (qemu-io's aio_read and aio_write commands can be used to trigger

I fixed the deadlock - turned out to be an actual bug in the nbd plugin (see patch 8). I changed the command syntax so it's now: nbdkit nbd command=qemu arg=-f arg=qcow2 arg=/path/to/disk.qcow2 Nir wrote: 18:08 < nsoffer> rwmjones: regarding the nbd proxy patches, did you have specific flow that help us? 18:08 < nsoffer> rwmjones: or this is just a way to support qcow2 in the

The first four patches are fairly routine clean up and can be reviewed/applied on their own. The fifth patch is problematic as described below. At the moment if you want to proxy through to qemu-nbd (eg. for handling qcow2 files) it's rather complicated and you end up having to manage the sockets and clean up yourself. However the library we use for the proxying supports a perfectly good

gcc 8 gripes (when using './configure --enable-gcc-warnings'): nbd.c: In function 'nbd_open': nbd.c:470:3: error: 'strncpy' specified bound 108 equals destination size [-Werror=stringop-truncation] strncpy (sock.sun_path, sockname, sizeof (sock.sun_path)); The warning is a false positive, given that we currently reject names >= sizeof(sock.sun_path), and thus we are

Another series on top of my exportname filter, marking off another todo bullet point. With this, you can now use the NBD plugin as a transparent passthrough of all export names served by the remote server in both directions (list advertisement server to client, and export name from client to server). Eric Blake (3): nbd: Implement .default_export, .export_description nbd: Add

When we first used libnbd (nbdkit 1.14), it was brand new, so we left in the ability to build a limited nbd plugin without the use of libnbd for comparison. But now that libnbd has had more time under its belt, and we have learned that our nbd-standalone code is not getting much testing, it's time to follow through with our plan to finish off the old code, and now build the nbd plugin only

With this, I am finally able to get the nbd plugin to do out-of-order responses to the client. Once this series goes in, we should be ready for Rich to cut a release. Eric Blake (4): nbd: Split reading into separate thread nbd: Protect writes with mutex nbd: Enable parallel handling tests: Test parallel nbd behavior plugins/nbd/nbd.c | 217

With new enough libnbd, we already support vsock by virtue of uri=; however, it's also nice to allow direct exposure of the nbd_connect_vsock() api. Signed-off-by: Eric Blake <eblake@redhat.com> --- As with commit 7ce9feef, there is no easy way to add testsuite coverage for this. plugins/nbd/nbdkit-nbd-plugin.pod | 30 +++++++++----- plugins/nbd/nbd.c | 65

I've documented a desire to do this for a while, time to actually follow through and support connecting as a client to a TCP server. Note that it is desirable to support the plugin connecting to an encrypted server over TCP, then exposing the raw data over a local Unix socket; that aspect requires yet more work, left for another day. But even allowing an old-style client to connect to an

With new enough libnbd, we already support vsock by virtue of uri=; however, it's also nice to allow direct exposure of the nbd_connect_vsock() api. Signed-off-by: Eric Blake <eblake@redhat.com> --- plugins/nbd/nbdkit-nbd-plugin.pod | 29 +++++++++++--- plugins/nbd/nbd.c | 63 ++++++++++++++++++++++++------- 2 files changed, 73 insertions(+), 19 deletions(-) diff --git

There are a few other issues that Coverity found, but I believe all can be ignored ... except one: We don't set umask anywhere inside nbdkit. Coverity complains that this is a problem where we create temporary files, since the result of mkstemp depends implicitly on the umask value. I think we might consider setting umask anyway (eg. to 022) just to make plugin behaviour more predictable.

I tried reproducing the testsuite failure on test-parallel-*.sh using the same machine Rich posted a log from, but did not quickly hit it on a loop of make -j20 check TESTS=test-parallel-{file,nbd}.sh But I still think this series can't hurt. Eric Blake (2): nbd: Don't advertise writes if nbdkit is readonly tests: Make parallel tests more robust plugins/nbd/nbd.c | 2 ++

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket, and the transactions are serialized rather than

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket with no encryption, and the transactions are serialized

Updated based on other changes that have happened in the meantime: - rely more on cleanup.h (throughout) - split structured read for easier review (patch 2 and 3 were combined in v1) - rely on nbdkit not leaking a server's partial answer (patch 3) - add tests (patch 5) - other bug fixes I found while testing it - drop EOVERFLOW patch for now; it will be separate once upstream NBD protocol