similar to: [libnbd PATCH 0/2] fix hangs against nbdkit 1.2

I still need more soak time on testing, to see whether I can: a) reproduce the hang with patch 2 not applied b) feel confident that patch 2 is sufficient to fix the race, or else determine that I also need to augment the loop condition in the reader thread to additionally break out of the loop when the pipe-to-self sees EOF even when nbd_aio_is_dead() has not yet been satisfied I'm also

Checking for poll(POLLIN) only wakes us up when the server sends more bytes over the wire. But the way gnutls is implemented, it reads as many non-blocking bytes as possible from the wire before then parsing where the encoded message boundaries lie within those bytes. As a result, we may already have the server's next reply in memory without needing to block on the server sending yet more

On Fri, Mar 27, 2020 at 05:33:28PM -0500, Eric Blake wrote: > We have been seeing sporadic hangs on test-nbd-tls-psk.sh, where even > though the client to the 'nbdkit nbd' process has cleanly exited, > things are stalled in .close where nbd is trying to pthread_join() the > reader thread, while the reader thread is itself blocked on a poll() > that will never make additional

Commit 9e6d990f exposed a pre-existing deadlock between the nbd plugin as client and parallel nbdkit as server. Prior to that commit, the deadlock was "resolved" because we unloaded the .so in parallel to a .close callback that never completed (yes, it's nasty that it usually? let the testsuite pass), but now we correctly refuse to unload a plugin that has not returned from .close,

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

We have been seeing sporadic hangs on test-nbd-tls-psk.sh and test-nbd-tls.sh (on my machine, running those two in a loop with commits 0a76cae4 and 09e34ba2 reverted would fail within 100 attempts), where even though the client to the 'nbdkit nbd' process has cleanly exited, things are stalled in .close where nbd is trying to pthread_join() the reader thread, while the reader thread is

ThreadSanitizer [1] pointed out that in the nbd plugin, nbd_close() can attempt close() in the main thread while the worker thread is still attempting to start a read(). Normally, if the read() loses the race, it will get a harmless EBADF that exits the worker thread (which is what we want, as we are closing the connection anyway); but if another connection happens to start in that window, we

We have been seeing sporadic hangs on test-nbd-tls-psk.sh, where even though the client to the 'nbdkit nbd' process has cleanly exited, things are stalled in .close where nbd is trying to pthread_join() the reader thread, while the reader thread is itself blocked on a poll() that will never make additional progress. Tracing the race is difficult: nbd_shutdown() sends NBD_CMD_DISC to the

On 3/30/20 2:59 PM, Eric Blake wrote: > nbdkit 1.2 as a server waits for read() to see EOF, even after the > client has sent NBD_CMD_DISC. That was fixed in mbdkit 1.4; and most > modern NBD servers are smarter than this (they close() the write end > of their traffic soon after NBD_CMD_DISC). But it's easy enough to > revert nbdkit commit c70616f8 to get back to a server with

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):

As mentioned in commits 176fc4ea and 609c25f0, our original plan in adding a flags argument to nbd_shutdown was to let us specify different behaviors at the libnbd level, rather than NBD protocol flags (for that, the user has nbd_aio_disconnect). But when we later parameterized OFlags to accept various bitmasks (commit f891340b), we failed to mark nbd_shutdown as using a different bitmask than

This is the bare minimum needed to allow the user to take control over the rest of option negotiating. This patch adds several new API: nbd_set_opt_mode() - called during Created to enable the new mode nbd_get_opt_mode() - query whether opt mode is enabled nbd_opt_go() - used in Negotiating state to attempt to use export nbd_opt_abort() - used in Negotiating state to skip Connected state

Necessary when writing a client that wants to avoid unnecessary EINVAL errors from sending unaligned requests. At some point, we may want to add synchronous convenience API wrappers that do request splitting or read-modify-write to obey server constraints while still appearing to the library client as accepting any possible request. But such a wrapper should only be synchronous and not copied to

Having problems with STI. Names below have been changed to protect the innocent. Imagine a system that allows someone to apply for one of two different types of school scholarships. Each scholarship goes through a different review process, represented by a state machine (this is not a state machine question). So there are two state machine classes that differ slightly and subclass a generic

In v2: ack'ed preliminary patches have been pushed, and I've added a lot of testsuite coverage as well as putting the new API to use in nbdinfo. Eric Blake (2): api: Add nbd_opt_list_meta_context info: List available meta-contexts lib/internal.h | 1 + generator/API.ml | 84 ++++++++-

Proposal for new APIs to list exports. The general shape of the API can probably best be seen from the examples/list-exports.c example. Rich.

v1 was posted here: https://www.redhat.com/archives/libguestfs/2019-September/thread.html#00337 v2: - Drop the first patch. - Hopefully fix the multiple issues with fork-safety and general behaviour on error paths. Note this requires execvpe for which there seems to be no equivalent on FreeBSD, except some kind of tedious path parsing (but can we assign to environ?) Rich.

This is a subset of my v2 posting, but limited to just the NBD_OPT_LIST handling. The biggest change since v2 is the addition of added unit testing in all four language bindings (C, python, ocaml, golang). The tests require nbdkit built from git on PATH, and may not be entirely idiomatic, but I at least validated that they catch issues (for example, adding an exit statement near the end of the

This was v3 patch 2/22, reworked to address the confusion about how a structured reply header is read in two pieces before getting to the payload portion. I'm still working on rebasing the rest of my v3 series (patches 1, 3-22) from other comments given, but this seemed independent enough that it's worth posting now rather than holding it up for the rest of the series. Eric Blake (4):

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit