similar to: [PATCH nbdkit v2] common: Introduce round up, down; and divide round

The partitioning plugin patch is the same (except for rebasing). However I have changed the first two patches based on feedback received. In particular this fixes a very serious bug found by Eric Blake in the current truncate filter. Rich.

nbdkit partitioning boot.img swap.img root.img ... creates a virtual disk by adding a partition table. In ancient times Xen used to do this. Rich.

These are used at various places in the code already. This refactoring simply moves them to a common header file and should have no other effect. --- common/include/rounding.h | 59 +++++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 3 +- filters/cache/cache.c | 2 +- filters/cow/Makefile.am | 3 +- filters/cow/cow.c | 2 +- filters/truncate/truncate.c

This refactoring also fixes a bug spotted by Eric Blake: nbdkit --filter truncate pattern size=5G round-up=512 results in a 1G virtual disk. --- common/include/rounding.h | 58 ++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 3 +- filters/cache/cache.c | 2 +- filters/cow/Makefile.am | 3 +- filters/cow/cow.c | 2 +- filters/truncate/truncate.c | 5

While working on adding assertions to pthread_mutex_lock calls, I noticed that the truncate filter's use of mutex didn't really protect us, and isn't really necessary. Cleaning that up also spotted a couple of other potential cleanups. Eric Blake (4): filters: Drop useless .open callbacks truncate: Fix corruption when plugin changes per-connection size truncate: Test for safe

On 9/17/18 3:38 PM, Richard W.M. Jones wrote: > These are used at various places in the code already. This > refactoring simply moves them to a common header file and should have > no other effect. > --- > common/include/rounding.h | 59 +++++++++++++++++++++++++++++++++++++ > filters/cache/Makefile.am | 3 +- > filters/cache/cache.c | 2 +- >

I have dropped the map filter from this series for now while I try to get it working. However I think the truncate filter is in a good shape. This incorporates all feedback from Eric's review. Also there are three small fixes to the filter code, all revealed when I was testing using multiple filters which we'd not done much of before. Rich.

It is possible for the round-up parameter to cause our desired size to exceed 2**63-1. But this error message is cryptic: $ ./nbdkit -f --filter=truncate pattern $(((1<<63)-1)) round-up=1m & $ qemu-nbd --list nbdkit: pattern[1]: error: .get_size function returned invalid value (-9223372036854775808) qemu-nbd: Failed to read initial magic: Unexpected end-of-file before all bytes were

With these patches, a majority of tests pass. The notable things which are still broken: - Because FreeBSD links /home -> /usr/home, $(pwd) gives a different result from realpath(2). Therefore some tests which implicitly rely on (eg) a plugin which calls nbdkit_realpath internally and then checking that path against $(pwd) fail. - Shebangs (#!) don't seem to work the same way

I do have a question about whether patch 2 is right, or whether I've exposed a bigger problem in the truncate (and possibly other) filter, but the rest seem fairly straightforward. Eric Blake (4): server: Check for pthread lock failures truncate: Factor out reading real_size under mutex plugins: Check for mutex failures filters: Check for mutex failures filters/cache/cache.c

On 08/01/2018 06:10 AM, Richard W.M. Jones wrote: > This can truncate, extend, or round up/down to a multiple. > --- > common-rules.mk | 3 +- > configure.ac | 1 + > filters/offset/nbdkit-offset-filter.pod | 7 +- > filters/partition/nbdkit-partition-filter.pod | 1 + >

This is exactly the same as v1: https://www.redhat.com/archives/libguestfs/2018-December/msg00004.html except that it now frees the bitmap on unload (which the old code did not - there was always a memory leak). Rich.

I have some patches I'm working on to fix the cache filter. However this is a prelude. It should be simply pure refactoring. All tests pass still. Rich.

Being able to programmatically force nbdkit to be less parallel can be useful during testing. I was less sure about patch 3, but if you like it, I'm inclined to instead squash it into patch 1. This patch is written to apply after my NBD_CMD_CACHE work (since I touched the nocache filter); but can be rearranged if we think this series should go in first while that one undergoes any adjustments

The truncate filter tried to be careful to lock access to setting or reading the real_size variable learned from calling next_ops->get_size, in anticipation of not behaving incorrectly if the NBD protocol makes dynamic resize supported, and where the global variable could serve as a cache rather than having to always call next_ops->get_size to recompute things. However, nbdkit-plugin.pod

When using a filter that rounds up to alignment boundaries, the tail bytes of the plugin are difficult to access correctly. Rather than duplicating lots of code already in the truncate filter, it's easier to just make the other filters default to rounding down and add doc links on how to round up instead. Eric Blake (2): blocksize: Lift restriction against 0-size file cache, cow: Round

This patch series proposes two new filters. * truncate: This can truncate, extend, round up or round down the size of a plugin/device. A typical usage is to fix the qemu problem that it can only handle devices which are a multiple of 512-bytes: nbdkit --filter=truncate random size=500 round-up=512 This will serve a virtual device with size 512 bytes. Reading from the last 12 bytes will

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order

v2: https://www.redhat.com/archives/libguestfs/2018-December/msg00039.html v2 -> v3: - Fix all the issues raised in Eric's review. - Precompute some numbers to make the calculations easier. - Calculations now use bitshifts and masks in preference to division and modulo. - Clear existing bits before setting (which fixes a bug in the cache filter). Rich.

--- common/include/isaligned.h | 11 +++++------ plugins/file/file.c | 4 ++-- plugins/vddk/vddk.c | 8 ++++---- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/common/include/isaligned.h b/common/include/isaligned.h index e693820..81ce8a7 100644 --- a/common/include/isaligned.h +++ b/common/include/isaligned.h @@ -36,16 +36,15 @@ #include <assert.h>