similar to: [PATCH nbdkit 0/4] Add truncate and map filters.

I have dropped the map filter from this series for now while I try to get it working. However I think the truncate filter is in a good shape. This incorporates all feedback from Eric's review. Also there are three small fixes to the filter code, all revealed when I was testing using multiple filters which we'd not done much of before. Rich.

While working on adding assertions to pthread_mutex_lock calls, I noticed that the truncate filter's use of mutex didn't really protect us, and isn't really necessary. Cleaning that up also spotted a couple of other potential cleanups. Eric Blake (4): filters: Drop useless .open callbacks truncate: Fix corruption when plugin changes per-connection size truncate: Test for safe

Serve an arbitrary map of regions of the underlying plugin. --- common-rules.mk | 1 + configure.ac | 1 + filters/map/Makefile.am | 61 +++ filters/map/map.c | 256 +++++++++ filters/map/maptype.c | 493 ++++++++++++++++++ filters/map/maptype.h

I do have a question about whether patch 2 is right, or whether I've exposed a bigger problem in the truncate (and possibly other) filter, but the rest seem fairly straightforward. Eric Blake (4): server: Check for pthread lock failures truncate: Factor out reading real_size under mutex plugins: Check for mutex failures filters: Check for mutex failures filters/cache/cache.c

When querying the extents of the underlying plugin, we should only translate the starting offset, and let the plugin report for at least as many bytes as our range permits. Otherwise, short-changing the range causes bad behavior such as returning 0 extents, or even failing the creation of an extents tracker: $ cat script case "$1" in get_size) echo 1m;; can_extents) ;;

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

The man page claims both offset and range are optional (matching the code), but the --help text claims offset is mandatory, and the comment to the no-op offset_config_complete claims we require both parameters. We did not check for an offset larger than the underlying size when there was no range, and even when there is a range, we were not careful about integer overflow (offset=5E range=5E

v1 was here: https://www.redhat.com/archives/libguestfs/2018-September/msg00057.html v2: - Fix tab vs spaces in configure.ac. - To generate list of plugins, use printf instead of xargs. - Use 'source ./functions.sh' instead of 'source functions'. - functions.sh: Consistent quoting in foreach_plugin function. - functions.sh: Change the contract of start_nbdkit so it

With these patches, a majority of tests pass. The notable things which are still broken: - Because FreeBSD links /home -> /usr/home, $(pwd) gives a different result from realpath(2). Therefore some tests which implicitly rely on (eg) a plugin which calls nbdkit_realpath internally and then checking that path against $(pwd) fail. - Shebangs (#!) don't seem to work the same way

v1 was here: https://www.redhat.com/archives/libguestfs/2019-January/thread.html#00198 For v2 I changed most existing prerequisite tests to use the new mechanism. I only changed simple tests. There are a few more complex tests that don't fit the “requires model” and those are not changed. I normalized qemu-io/qemu-img testing to always use the --version flag, where previously we used a mix

Combine much common code into tests/functions.sh. Patch 1: Preparation for patch 3. Patch 2: Fix a long-standing bug in how man pages links are generated. Patch 3: Common code for iterating a test function over every plugin. Patch 4: Common code for starting nbdkit in a test and waiting for the PID file to appear. This is the largest and most complex of the patches but is

Rebased filters patch. Requires current git master + the locks / thread model fix (https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html) So a few changes here since last time: The "introduce filters" and "implement filters" patches are squashed together. I introduced a concept of .prepare and .finalize. These run before and after the data serving phase

There's more that can be done (in particular, use of CLEANUP_FREE), but this is enough to at least see if I'm on the right track. I couldn't figure out an obvious difference between common/include and common/utils, but it looks like the former is for things that are inlineable via .h only, while the latter is when you need to link in a convenience library, so this landed in the

Simpler, and including tests. Rich.

For review only, this needs some clean up and more tests. My eyes are going cross-eyed looking at the calculate_offset_of_entry function, so time to take a break ... Rich.

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00199.html v2: - Adds a fairly simple yet comprehensive test using sh plugin. - Rebase and retest. Patch 1 is a misc patch not really related to the series. Rich.

Since v1: - rework .can_cache to be tri-state, with default of no advertisement (ripple effect through other patches) - add a lot more patches in order to round out filter support And in the meantime, Rich pushed NBD_CMD_CACHE support into libnbd, so in theory we now have a way to test cache commands through the entire stack. Eric Blake (24): server: Internal hooks for implementing

This is still tentative and needs a lot of work, but: - partition filter works, supporting MBR & GPT - prepare and finalize methods fixed - open method can now be changed (allowing readonly flag to be modified) - thread_model can be limited I believe I made most of the changes which were previously suggested in email. I think the only one I didn't was preventing inclusion of both

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

v2: - Fixes a number of bugs in corner cases. - Uses a 1M block size to fetch from the underlying plugin. This improves performance considerably. I also tested this much more thoroughly and can't find any more bugs. Rich.