similar to: [nbdkit PATCH 0/5] More retry fixes

v1 was here: https://www.redhat.com/archives/libguestfs/2020-February/msg00081.html v2 replaces struct connection *conn = GET_CONN; with GET_CONN; which sets conn implicitly and asserts that it is non-NULL. If we actually want to test if conn is non-NULL or behave differently, then you must use threadlocal_get_conn() instead, and some existing uses do that. Rich.

The third patch is a large but mechanical change which gets rid of passing around struct connection * entirely within the server, preferring instead to reference the connection through thread-local storage. I hope this is a gateway to simplifying other parts of the code. Rich.

I still don't have .prepare/.finalize working cleanly across reopen, but did find a nasty bug where a botched assertion means we failed to notice reads beyond EOF in both the xz and retry filter. Refactoring backend.c will make .finalize work easier. Eric Blake (5): xz: Avoid reading beyond EOF retry: Check size before transactions tests: Test retry when get_size values change

It's confusing to use the same terminology for a single backend as for the linked list of backends. In particular it's often not clear if we're calling the next backend or the whole chain of backends. --- server/internal.h | 14 ++++++++++-- server/connections.c | 20 ++++++++--------- server/locks.c | 2 +- server/main.c

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00199.html v2: - Adds a fairly simple yet comprehensive test using sh plugin. - Rebase and retest. Patch 1 is a misc patch not really related to the series. Rich.

This is a retry filter implementation as outlined here: https://www.redhat.com/archives/libguestfs/2019-September/msg00167.html It is only lightly tested. One way to test it is to try an SSH copy (see the commit message for patch 2/2), and in the middle of the copy kill the per-connection sshd on the remote machine. You will see that the copy recovers after a few seconds. Add the nbdkit -v

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

This is a subset of the last half of the larger 9-patch series. The uncontroversial first half of that series is pushed, but here, I tried to reduce the size of the patches by splitting out some of the more complex changes, so that the rest of the changes remaining in the series are more mechanical. In turn, it forced me to write timing tests, which let me spot another spot where we are wasting

It's easy to use the sh script to demonstrate that nbdkit is inefficiently calling into .get_size, .can_fua, and friends more than necessary. We've also commented on the list in the past that it would be nice to ensure that when filters call into next_ops, they are not violating constraints (as we've have to fix several bugs in the past where we did not have such checking to protect

On Thu, Oct 03, 2019 at 09:54:37PM -0500, Eric Blake wrote: > Although it is unusual for a plugin to shrink size on reopen, it is > not impossible. Failure to check our bounds could result in violating > assumptions in the plugin that all requests are in-bounds. Note that > if the plugin gains a larger size, we merely never access the new tail > of the file (when the NBD protocol

The immediate reason for this patch is to reduce the amount of debugging in virt-v2v with using the virt-v2v -v option (because this implies running nbdkit in verbose mode too). Most of the messages are datapath ones about pread/pwrite requests, and in fact as we've added more filters on top of nbdkit these messages have got more and more verbose. However they are not particularly

I'm still working on another set of patches to have reopen call .finalize/.prepare (so that another filter can safely appear between retry and the plugin), but for tonight, these are the patches I think are ready to go. Eric Blake (4): retry: Handle can_fua and can_fast_zero changes tests: Test retry with different fua/fast-zero flags server: Close backends if a filter's .open fails

The first one is the nastiest - it is an assertion failure caused by a spec-compliant client and introduced by our security fix that was released in 1.14.1. Eric Blake (4): server: Fix regression for NBD_OPT_INFO before NBD_OPT_GO server: Fix back-to-back SET_META_CONTEXT server: Forbid NUL in export and context names server: Fix OPT_GO on different export than SET_META_CONTEXT

Since commit 86fdb48c6a5362d66865493d9d2172166f99722e we have stored the connection object in thread-local storage. In this very large, but mostly mechanical change we stop passing the connection pointer around everywhere, and instead use the value stored in thread-local storage. This assumes a 1-1 mapping between the connection and the current thread which is true in *most* places.

On Thu, Sep 19, 2019 at 12:34:17PM +0100, Richard W.M. Jones wrote: > diff --git a/server/backend.c b/server/backend.c > index 8a434bd..b8c5742 100644 > --- a/server/backend.c > +++ b/server/backend.c > @@ -224,9 +224,17 @@ backend_valid_range (struct backend *b, struct connection *conn, > int > backend_reopen (struct backend *b, struct connection *conn, int readonly) >

I'm not sure that this fix is really correct. An alternate way I can think to fix this would be for the core server to maintain a readonly flag for each layer (instead of just per- server). You could also argue that our readonly test in server/connections.c:compute_eflags is wrong and/or that the implementations of server/plugins.c:plugin_can_write and server/filters.c:filter_can_write have

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

Rebased filters patch. Requires current git master + the locks / thread model fix (https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html) So a few changes here since last time: The "introduce filters" and "implement filters" patches are squashed together. I introduced a concept of .prepare and .finalize. These run before and after the data serving phase

Applies on top of my pending series for the exportname filter, addressing one of the todo's in that cover letter. Eric Blake (2): filters: Add .export_description wrappers ext2: Supply .list_exports and .default_export filters/ext2/nbdkit-ext2-filter.pod | 3 +- tests/Makefile.am | 16 +++- filters/ext2/ext2.c | 125 +++++++++++++++++++---------

To allow filters to modify the export name as it passes through the layers this commit makes several changes: The filter .open callback now takes an extra parameter, the export name. This is always non-NULL (for oldstyle it is ""). This string has a short lifetime and filters that need to hang on to it must take a copy. The filter must pass the exportname parameter down to the next