similar to: [nbdkit PATCH] nbd: Drop nbd-standalone fallback

This conversion should be feature compatible with the standalone nbd code. Note that the use of libnbd makes the binary for this particular plugin fall under an LGPLv2+ license rather than BSD; but the source code in nbd.c remains BSD. A lot of code simply disappears, now that I'm no longer directly utilizing the NBD protocol files but relying on libnbd. Coordination between threads from

This conversion should be feature compatible with the standalone nbd code. Note that the use of libnbd makes the binary for this particular plugin fall under an LGPLv2+ license rather than BSD; but the source code in nbd.c remains BSD. A lot of code simply disappears, now that I'm no longer directly utilizing the NBD protocol files but relying on libnbd. Coordination between threads from

Time to enhance the nbd plugin to request structured replies from the server. For now, deal only with structured reads. The server can now return sparse reads, even though we need nbdkit version 3 before we can in turn return sparse reads back to the client. In general, we have to assume the server is malicious, and so we must sanity check that it sends replies we expect. Thus, we have a choice

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket with no encryption, and the transactions are serialized

The NBD spec was recently patched (nbd.git commit 7827f3ae and friends) to require NBD_OPT_GO for baseline interoperability, with the aim of fewer servers and clients falling back to NBD_OPT_EXPORT_NAME. And since nbdkit as server recently started supporting NBD_OPT_GO (commit f7dd9799), our nbd client as plugin should take advantage of it. This patch is a prerequisite to teaching the nbd plugin

In preparation for allowing interleaved response, refactor the nbd forwarder so that writes are still done from the thread handling the original request from the client, but all reads are done by a dedicated reader thread. Control between the two flags is gated by a mutex for storing the transaction information, coupled with a pipe for the reader thread to send the final status back to the

I ran some local testing on my Fedora 29 system via: $ ./nbdkit memory 1m $ for i in `seq 10`; do ./nbdkit -U - --filter=stats nbd \ statsappend=true statsfile=$file hostname=localhost port=10809 \ --run '~/libnbd/examples/threaded-reads-and-writes $unixsocket' done Pre-patch, the runs averaged 1.266s, 1.30E+08 bits/s Post-patch, the runs averaged 1.154s, 1.42E+08

This is a minimal implementation of an NBD forwarder; it lets us convert between old and newstyle connections (great if a client expects one style but the real server only provides the other), or add TLS safety on top of a server without having to rewrite that server. Right now, the real server is expected to live on a named Unix socket, and the transactions are serialized rather than

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd:

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros.

I've documented a desire to do this for a while, time to actually follow through and support connecting as a client to a TCP server. Note that it is desirable to support the plugin connecting to an encrypted server over TCP, then exposing the raw data over a local Unix socket; that aspect requires yet more work, left for another day. But even allowing an old-style client to connect to an

libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing. Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+, add tests to TLS usage which flushed out the need to turn relative pathnames into absolute, doc tweaks Now that the testsuite covers TLS and libnbd has been fixed to provide the things I found lacking when developing v2, I'm leaning towards pushing this on

On 3/8/19 4:04 AM, Richard W.M. Jones wrote: > This is about the simplest implementation of NBD Structured Replies > (SR) that we can do right now. > > It accepts NBD_OPT_STRUCTURED_REPLIES when negotiated by the client, > but only sends back the simplest possible SR when required to by > NBD_CMD_READ. The rest of the time it will send back simple replies > as before. We do

On 4/22/19 7:50 PM, Eric Blake wrote: > Time to enhance the nbd plugin to request structured replies from the > server. For now, deal only with structured reads. The server can now > return sparse reads, even though we need nbdkit version 3 before we > can in turn return sparse reads back to the client. > > In general, we have to assume the server is malicious, and so we must

Use new 'name_of_*' functions to offer a bit more details about messages being forwarded on to the remote server. Signed-off-by: Eric Blake <eblake@redhat.com> --- plugins/nbd/nbd.c | 8 +++++--- plugins/nbd/Makefile.am | 3 ++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index 672f35b..6bd8861 100644 ---

This is about the simplest implementation of NBD Structured Replies (SR) that we can do right now. It accepts NBD_OPT_STRUCTURED_REPLIES when negotiated by the client, but only sends back the simplest possible SR when required to by NBD_CMD_READ. The rest of the time it will send back simple replies as before. We do not modify the plugin API so plugins are unable to send complex SRs. Also we

On Fri, Dec 07, 2018 at 10:00:42AM -0600, Eric Blake wrote: > Use new 'name_of_*' functions to offer a bit more details about > messages being forwarded on to the remote server. > > Signed-off-by: Eric Blake <eblake@redhat.com> > --- > plugins/nbd/nbd.c | 8 +++++--- > plugins/nbd/Makefile.am | 3 ++- > 2 files changed, 7 insertions(+), 4 deletions(-)

Updated based on other changes that have happened in the meantime: - rely more on cleanup.h (throughout) - split structured read for easier review (patch 2 and 3 were combined in v1) - rely on nbdkit not leaking a server's partial answer (patch 3) - add tests (patch 5) - other bug fixes I found while testing it - drop EOVERFLOW patch for now; it will be separate once upstream NBD protocol

I'm hoping to implement .extents for the nbd plugin; this is a prerequisite. I'm not sure about patch 3 - if we like it, I'll squash it to 2, if we don't, I think we are okay just dropping it. I'm also wondering if we have to worry about malicious plugins that don't populate the entire .pread buffer in an effort to get nbdkit to expose portions of the heap; my patch 7 loses

qemu-nbd defaults to permitting only a single NBD client. Of course, you can run qemu-nbd -t to work around it, but other servers may have similar restrictions, at which point nbdkit's nbd plugin can provide the automatic fanout to multiple clients via a single server link. Note that the shared=true parameter makes the previously-added retry=N parameter more useful, as it is much easier to