similar to: [nbdkit PATCH 0/4] Fix testsuite hang with nbd-stadalone

I tried reproducing the testsuite failure on test-parallel-*.sh using the same machine Rich posted a log from, but did not quickly hit it on a loop of make -j20 check TESTS=test-parallel-{file,nbd}.sh But I still think this series can't hurt. Eric Blake (2): nbd: Don't advertise writes if nbdkit is readonly tests: Make parallel tests more robust plugins/nbd/nbd.c | 2 ++

qemu-io 2.12 will be changing its default alignment to unknown servers so that it does read-modify-write for anything less than 512 bytes. If we implement NBD_OPT_GO, then we can keep qemu-io using 1-byte alignment; but until then, this breaks our parallel tests when using 1-byte alignment because they end up with more delays than expected (thanks to the read-modify-write). Revamp the tests to

Small reworking of tests to remove $QEMU_IO, making that consistent with other test tools, and to test IPv6 connections.

Currently we test for qemu-img, socat, ss, certtool, etc at run time, but we test for qemu-io at compile time (in ./configure). This commit removes this inconsistency. I would consider the opposite patch (which makes qemu-img etc tested at configure time). The main advantage of testing for these binaries at run time is that tests are not "silently" omitted. Instead tests with

This works OK on x86_64, but fails on our fast new Amberwing (aarch64) machine. I've attached the test-suite.log file, but I'm not very sure what's going wrong from that. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to

A combination of the work that both Rich and I have been doing lately, where filters use only the new API with flags on every command that the client can send over the wire (we can then add support for more flags in nbdkit without having to add new callbacks, as NBD adds more flags upstream). Eric Blake (4): protocol: Split flags from cmd field in requests backend: Pass flags argument through

Remove the need to use file= (and in future other) parameters for many plugins. eg. Using the file plugin becomes: nbdkit file disk.img Rich.

The first three patches are identical to: https://www.redhat.com/archives/libguestfs/2018-January/msg00079.html "[PATCH nbdkit v2 0/3] Refactor plugin_* functions into a backend" The rest of the patches add filters using the new filter API previously described here: https://www.redhat.com/archives/libguestfs/2018-January/msg00073.html This needs a lot more testing -- and tests --

v1 was here: https://www.redhat.com/archives/libguestfs/2018-September/msg00024.html v2: - As discussed in the patch review, tighten up the characters permitted in keys. - Update documentation to note that relative paths can be made safe by prefixing with ./ and absolute paths do not need any extra steps. - I pushed patch 1/6 from the v1 series since it was just a trivial

v1: https://www.redhat.com/archives/libguestfs/2018-September/msg00024.html v2: https://www.redhat.com/archives/libguestfs/2018-September/msg00034.html v3: - Fixed is_config_key zero length test. - Fixed is_config_key so it uses strspn and is not O(n^2). - Changed >= 1.7 to >= 1.8 in the documentation. Rich.

Rebased filters patch. Requires current git master + the locks / thread model fix (https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html) So a few changes here since last time: The "introduce filters" and "implement filters" patches are squashed together. I introduced a concept of .prepare and .finalize. These run before and after the data serving phase

This is still tentative and needs a lot of work, but: - partition filter works, supporting MBR & GPT - prepare and finalize methods fixed - open method can now be changed (allowing readonly flag to be modified) - thread_model can be limited I believe I made most of the changes which were previously suggested in email. I think the only one I didn't was preventing inclusion of both

This is a major rewrite compared to my v1 series, where I've tried a lot harder to ensure that we still accommodate building on Haiku (although I have not actually yet fired up a Haiku VM to try it for myself). I also managed to make the sh plugin fully parallel, on capable platforms. See also my question on patch 10 on whether I've picked the best naming convention. Eric Blake (17):

On 01/17/2018 02:53 PM, Richard W.M. Jones wrote: > Previously the file plugin supported ‘rdelay’ and ‘wdelay’ parameters > for injecting delays (for testing) into read and write requests. This > moves the functionality to a new delay filter so that it can be used > with any plugin. > --- > +/* Write data. */ > +static int > +delay_pwrite (struct nbdkit_next *next, void

With this, I am finally able to get the nbd plugin to do out-of-order responses to the client. Once this series goes in, we should be ready for Rich to cut a release. Eric Blake (4): nbd: Split reading into separate thread nbd: Protect writes with mutex nbd: Enable parallel handling tests: Test parallel nbd behavior plugins/nbd/nbd.c | 217

I've posted some of these patches or ideas before; but now I'm confident enough with the series that it should be ready to push; at any rate, I can now run test-socket-activation in a tight loop without triggering any crashes or hangs. With this in place, I'm going back to work on making the nbd forwarder wort with the parallel thread model. Eric Blake (8): sockets: Use

On Wed, Jan 17, 2018 at 08:01:35PM -0600, Eric Blake wrote: > Actually, thinking about this more: > > When I added zero support, I documented in commit ac3f294a that for the > file plugin, wdelay is indeed doubled on systems lacking efficient zero > support. But the fallback for handling EOPNOTSUPP is currently done at > the plugins.c level (ie. it is part of next->zero()) -

This patch isn't complete (patch 6/6 isn't finished) so it's just for discussion, although it does compile and run. This introduces to nbdkit a concept of "filters" which can be placed in front of plugins to modify their behaviour. Some examples where you might use filters: * Serve a subset of the data, such as (offset, range) or a single partition from a disk image.

On 3/18/20 8:21 PM, Eric Blake wrote: > In commit c306fa93ab and neighbors (v1.15.1), a concerted effort went > into caching the results of .can_FOO callbacks, with commit messages > demonstrating that a plugin with a slow callback should not have that > delay magnified multiple times. But nothing was added to the > testsuite at the time, and with the sh and eval plugins, we still

If nbdkit has a bug (such as the nbd-standalone bug fixed in the previous commit), qemu-io ends up waiting forever rather than realizing that if the server disappears unexpectedly then qemu-io should quit. So add timeouts so the testsuite will flag the problem instead of hang (tested by reordering this commit before the previous). It's trickier than I expected: from the command line,