similar to: [nbdkit PATCH 0/2] Let nbd plugin connect to TCP socket

I got annoyed by qemu-nbd's default of only allowing a single connection; combine that with nbdkit's nbd plugin, and even 'qemu-nbd --list' of nbdkit counts as the single connection and immediately hangs up. If we introduce a shared mode, then 'qemu-nbd --list' can connect as many times as it wants without killing the original qemu-nbd wrapped by nbdkit. But this in turn

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd:

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros.

libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing. Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+, add tests to TLS usage which flushed out the need to turn relative pathnames into absolute, doc tweaks Now that the testsuite covers TLS and libnbd has been fixed to provide the things I found lacking when developing v2, I'm leaning towards pushing this on

I'm still working on the interleaving (and Rich reminded me on IRC that we still don't have THREAD_MODEL_PARALLEL working anywhere yet, anyways). Since nbdkit doesn't really have a parallel plugin yet, my testing on that front will have to use qemu-nbd as the original server, as well as qemu-io as the driver (qemu-io's aio_read and aio_write commands can be used to trigger

gcc 8 gripes (when using './configure --enable-gcc-warnings'): nbd.c: In function 'nbd_open': nbd.c:470:3: error: 'strncpy' specified bound 108 equals destination size [-Werror=stringop-truncation] strncpy (sock.sun_path, sockname, sizeof (sock.sun_path)); The warning is a false positive, given that we currently reject names >= sizeof(sock.sun_path), and thus we are

I've documented a desire to do this for a while, time to actually follow through and support connecting as a client to a TCP server. Note that it is desirable to support the plugin connecting to an encrypted server over TCP, then exposing the raw data over a local Unix socket; that aspect requires yet more work, left for another day. But even allowing an old-style client to connect to an

There are a few other issues that Coverity found, but I believe all can be ignored ... except one: We don't set umask anywhere inside nbdkit. Coverity complains that this is a problem where we create temporary files, since the result of mkstemp depends implicitly on the umask value. I think we might consider setting umask anyway (eg. to 022) just to make plugin behaviour more predictable.

With this, I am finally able to get the nbd plugin to do out-of-order responses to the client. Once this series goes in, we should be ready for Rich to cut a release. Eric Blake (4): nbd: Split reading into separate thread nbd: Protect writes with mutex nbd: Enable parallel handling tests: Test parallel nbd behavior plugins/nbd/nbd.c | 217

With new enough libnbd, we already support vsock by virtue of uri=; however, it's also nice to allow direct exposure of the nbd_connect_vsock() api. Signed-off-by: Eric Blake <eblake@redhat.com> --- As with commit 7ce9feef, there is no easy way to add testsuite coverage for this. plugins/nbd/nbdkit-nbd-plugin.pod | 30 +++++++++----- plugins/nbd/nbd.c | 65

The first four patches are fairly routine clean up and can be reviewed/applied on their own. The fifth patch is problematic as described below. At the moment if you want to proxy through to qemu-nbd (eg. for handling qcow2 files) it's rather complicated and you end up having to manage the sockets and clean up yourself. However the library we use for the proxying supports a perfectly good

With new enough libnbd, we already support vsock by virtue of uri=; however, it's also nice to allow direct exposure of the nbd_connect_vsock() api. Signed-off-by: Eric Blake <eblake@redhat.com> --- plugins/nbd/nbdkit-nbd-plugin.pod | 29 +++++++++++--- plugins/nbd/nbd.c | 63 ++++++++++++++++++++++++------- 2 files changed, 73 insertions(+), 19 deletions(-) diff --git

I fixed the deadlock - turned out to be an actual bug in the nbd plugin (see patch 8). I changed the command syntax so it's now: nbdkit nbd command=qemu arg=-f arg=qcow2 arg=/path/to/disk.qcow2 Nir wrote: 18:08 < nsoffer> rwmjones: regarding the nbd proxy patches, did you have specific flow that help us? 18:08 < nsoffer> rwmjones: or this is just a way to support qcow2 in the

qemu-nbd defaults to permitting only a single NBD client. Of course, you can run qemu-nbd -t to work around it, but other servers may have similar restrictions, at which point nbdkit's nbd plugin can provide the automatic fanout to multiple clients via a single server link. Note that the shared=true parameter makes the previously-added retry=N parameter more useful, as it is much easier to

More nbd-protocol.h improvements Eric Blake (2): common/protocol: Switch nbdmagic to uint64_t common/protocol: Declare additional constants common/protocol/nbd-protocol.h | 16 ++++++++++------ server/protocol-handshake-newstyle.c | 2 +- server/protocol-handshake-oldstyle.c | 2 +- plugins/nbd/nbd-standalone.c | 2 +- tests/test-layers.c | 2 +- 5 files

Updated based on other changes that have happened in the meantime: - rely more on cleanup.h (throughout) - split structured read for easier review (patch 2 and 3 were combined in v1) - rely on nbdkit not leaking a server's partial answer (patch 3) - add tests (patch 5) - other bug fixes I found while testing it - drop EOVERFLOW patch for now; it will be separate once upstream NBD protocol

Another series on top of my exportname filter, marking off another todo bullet point. With this, you can now use the NBD plugin as a transparent passthrough of all export names served by the remote server in both directions (list advertisement server to client, and export name from client to server). Eric Blake (3): nbd: Implement .default_export, .export_description nbd: Add

When we first used libnbd (nbdkit 1.14), it was brand new, so we left in the ability to build a limited nbd plugin without the use of libnbd for comparison. But now that libnbd has had more time under its belt, and we have learned that our nbd-standalone code is not getting much testing, it's time to follow through with our plan to finish off the old code, and now build the nbd plugin only

I'm hoping to implement .extents for the nbd plugin; this is a prerequisite. I'm not sure about patch 3 - if we like it, I'll squash it to 2, if we don't, I think we are okay just dropping it. I'm also wondering if we have to worry about malicious plugins that don't populate the entire .pread buffer in an effort to get nbdkit to expose portions of the heap; my patch 7 loses

Experimental and only very lightly tested so far. Rich.