search for: recv_request_send_repli

I want to make my nbd forwarding plugin fully parallel - but to do that, I first need to make nbdkit itself fully parallel ;) With this series, I was finally able to demonstrate out-of-order responses when using qemu-io (which is great at sending back-to-back requests prior to waiting for responses) coupled with the nbd file plugin (which has a great feature of rdelay and wdelay, to make it

In preparation for parallel processing, we need to be sure that two threads belonging to the same connection cannot interleave their I/O except at message boundaries. Add a mutex around all reads and writes that must occur as a group (for now, there is no contention for either mutex). Signed-off-by: Eric Blake <eblake@redhat.com> --- src/connections.c | 27 +++++++++++++++++++++++++-- 1

On Thu, Nov 16, 2017 at 09:26:54PM -0600, Eric Blake wrote: > In preparation for parallel processing, we need to be sure that > two threads belonging to the same connection cannot interleave > their I/O except at message boundaries. Add a mutex around > all reads and writes that must occur as a group (for now, there > is no contention for either mutex). > > Signed-off-by:

Hi Eric, There's a memory leak in the nbd client. The message (below) is not very useful because somehow debuginfo is missing in the plugin. However it's easily reproducible by doing: make check-valgrind TESTS=test-nbd I tried the attached patch to fix what I thought was the bug, but sadly the fix doesn't work for me :-( So I guess something else is going on, but it does look as

I've posted some of these patches or ideas before; but now I'm confident enough with the series that it should be ready to push; at any rate, I can now run test-socket-activation in a tight loop without triggering any crashes or hangs. With this in place, I'm going back to work on making the nbd forwarder wort with the parallel thread model. Eric Blake (8): sockets: Use

The concern is a client is blocked while processing a request. The nbdkit server design requires a thread per request being processed regardless of the number of connections or clients. We want to run 1000's of requests in parallel without needing a thread at nbdkit layer per request in flight. Our plugin layer is built around boost asio and a few threads in a worker pool running an io

When facing a malicious client that is sending bogus NBD_CMD_READ, we should make sure that we never end up in a situation where we could try to treat the tail from a command that we diagnosed as bad as being further commands. Eric Blake (2): connections: Report mid-message EOF as fatal connections: Hang up early on insanely large WRITE requests src/connections.c | 35

With some crufty sed scripts we can generate functions that map from NBD protocol flags (eg. NBD_CMD_READ) to strings ("NBD_CMD_READ"). This works on GNU sed and with FreeBSD, also with GNU sed's --posix option, so I guess the sed code is POSIX-compatible. Rich.

We had several inconsistencies from the NBD spec when diagnosing bad client messages: - FLUSH is not generally forbidden on a read-only export (so failing with EPERM is wrong) [meanwhile, if we don't advertise flush because plugin_can_flush() fails, then rejecting with EINVAL is still okay] - returning EPERM (aka EROFS) for read-only exports should probably take precedence over anything else -

Commit 9e6d990f exposed a pre-existing deadlock between the nbd plugin as client and parallel nbdkit as server. Prior to that commit, the deadlock was "resolved" because we unloaded the .so in parallel to a .close callback that never completed (yes, it's nasty that it usually? let the testsuite pass), but now we correctly refuse to unload a plugin that has not returned from .close,

This is about the simplest implementation of NBD Structured Replies (SR) that we can do right now. It accepts NBD_OPT_STRUCTURED_REPLIES when negotiated by the client, but only sends back the simplest possible SR when required to by NBD_CMD_READ. The rest of the time it will send back simple replies as before. We do not modify the plugin API so plugins are unable to send complex SRs. Also we

Reject rather than silently ignoring unknown client request flags. Signed-off-by: Eric Blake <eblake@redhat.com> --- src/connections.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/connections.c b/src/connections.c index 16c6584..44b7530 100644 --- a/src/connections.c +++ b/src/connections.c @@ -546,6 +546,13 @@ validate_request (struct connection *conn,

According to the NBD spec, the server should close the connection rather than replying to NBD_CMD_DISC (after flushing any pending replies to earlier commands), and a compliant client should not send any data after that command. However, when nbdkit is running multithreaded, we already have multiple threads competing for a read lock, and each of those threads would try to read() from the socket,

Fixes for various issues found while implementing my nbd forwarder plugin. I'm okay if you choose to take some but not others; the most important one is patch 3 which fixes a protocol violation that makes it impossible for a client to try and recover from EIO failures over a partially-flaky source block device. Eric Blake (3): maint: Add emacs hint file maint: Add NBDKIT_GDB support to

On 3/8/19 4:04 AM, Richard W.M. Jones wrote: > This is about the simplest implementation of NBD Structured Replies > (SR) that we can do right now. > > It accepts NBD_OPT_STRUCTURED_REPLIES when negotiated by the client, > but only sends back the simplest possible SR when required to by > NBD_CMD_READ. The rest of the time it will send back simple replies > as before. We do

----- Forwarded message ----- Date: Sat, 18 Feb 2017 22:21:19 -0500 Subject: nbdkit async Hello, Hope this is the right person to contact regarding nbdkit design. I have a high latency massively parallel device that I am currently implementing as an nbdkit plugin in c++ and have run into some design limitations due to the synchronous callback interface nbdkit requires. Nbdkit is currently

Mostly code motion. --- src/Makefile.am | 1 + src/connections.c | 14 +++---- src/internal.h | 14 ++++--- src/locks.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/plugins.c | 77 +++++------------------------------- 5 files changed, 142 insertions(+), 79 deletions(-) diff --git a/src/Makefile.am b/src/Makefile.am index 12b9043..1f05eab 100644 ---

Mostly code motion. --- src/Makefile.am | 1 + src/connections.c | 14 +++---- src/internal.h | 14 ++++--- src/locks.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/plugins.c | 77 +++++------------------------------- 5 files changed, 142 insertions(+), 79 deletions(-) diff --git a/src/Makefile.am b/src/Makefile.am index 12b9043..1f05eab 100644 ---

It's finally time to implement one of the TODO items: we want to support a thread pool of parallel readers from the client, in order to allow multiple in-flight operations with potential out-of-order completion. We also need at least one plugin that supports parallel processing for testing the option; the file plugin fits the bill. Add and document a new command line option, -t/--threads=N,

On Wed, Jan 31, 2018 at 09:26:37PM -0600, Eric Blake wrote: > Previously, we let a plugin set an error in either thread-local > storage (nbdkit_set_error()) or errno, then connections.c would > decode which error to use. But with filters in the mix, it is > very difficult for a filter to know what error was set by the > plugin (particularly since nbdkit_set_error() has no public