similar to: [PATCH nbdkit PRELIMINARY] file: Move file operators to a new fileops mini-library

Needs some work still, see in particular the commit message for patch 3. Rich.

This is the third version to support efficient zero for block devices on older kernels (e.g. RHEL 7.5), and file systems that do not support yet FALLOC_FS_ZERO_RANGE (e.g. NFS 4.2). Changes since v2: - Revert file_can_trim change, since it is too late to change the value after negotiation. Changing the capability dinamically may be useful internally, but it should be done via other means. -

This version addresses comments on v3. Changes since v3: - Finally got spacing right (Eric) - Reorder includes (Richard) - Return 0 or -1 instead of r (Richard) - Add common/include/isaligned.h to Makefile.am (Richard) v3 was here: https://www.redhat.com/archives/libguestfs/2018-August/msg00177.html Nir Soffer (4): file: Avoid unsupported fallocate() calls file: Support zero without

This version addresses some of the comments on v2. Changes since v2: - file_zero: Add missing space in function call - is_aligned: Assert that align is indeed a power of 2 - Spelling in commit message Not changed: - Eric commented that spacing was off: https://www.redhat.com/archives/libguestfs/2018-August/msg00113.html but I could not find anything wrong. - Eric asked if ioctl.h will cause

This is the second version to support efficient zero for block devices on older kernels (e.g. RHEL 7.5), and file systems that do not support yet FALLOC_FS_ZERO_RANGE (e.g. NFS 4.2). Changes since v1: - Split to smaller patches - Skip linux only includes on other systems - Skip code using BLKZEROOUT if the macro is not defined - Try BLKZEROOUT only if the offset and count are aligned to device

Writing "file-like" plugins is hard because you have to implement your own .zero, .trim, .extents, etc, and that is very complicated. However implementations of these functions already exist in the file plugin. By factoring out the file plugin into a separate "fileops" mini-library we can reuse these implementations in other plugins. This refactoring commit creates a new

If we may not trim, we tried ZERO_RANGE, but this is not well supported yet, for example it is not available on NFS 4.2. ZERO_RANGE and PUNCH_HOLE are supported now on block devices, but not on RHRL 7, so we fallback to slow manual zeroing there. Change the logic to support block devices on RHEL 7, and file systems that do not support ZERO_RANGE. The new logic: - If we may trim, try PUNCH_HOLE -

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that

On 08/02/2018 02:05 PM, Nir Soffer wrote: > When using file systems not supporting ZERO_RANGE (e.g. NFS 4.2) or > block device on kernel < 4.9, we used to call fallocate() for every > zero, fail with EOPNOTSUPP, and fallback to manual zeroing. When > trimming, we used to try unsupported fallocate() on every call. > > Change file handle to remember if punching holes or

For existing commits, this is almost identical to v1, except that I updated some commit messages and reordered the commits in a somewhat more logical sequence. The main changes are the extra commits: [06/11] plugins: Return NBD_FLAG_CAN_MULTI_CONN from some readonly plugins. - Readonly plugins that can set the flag unconditionally. [09/11] partitioning: Return NBD_FLAG_CAN_MULTI_CONN. [10/11]

Unfinished (needs tests). This is my attempt to make a "remote tmpfs" plugin as outlined in this prior email: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html Although it would be possible to construct something a bit like this using existing plugins and filters (perhaps with some new features in those filters) I think it may be nicer to have a dedicated plugin for

I ran into these while trying to prepare patches to add NBD_CMD_FLAG_FAST_ZERO, which will expose a new NBD_ENOTSUP wire value. Eric Blake (2): plugins: Don't lose original error when emulating FUA plugins: Permit ENOTSUP as synonym for EOPNOTSUPP docs/nbdkit-filter.pod | 11 ++++++----- docs/nbdkit-plugin.pod | 12 +++++++----- plugins/file/file.c | 16 +++++++++++-----

On 3/20/19 5:11 PM, Richard W.M. Jones wrote: > This uses lseek SEEK_DATA/SEEK_HOLE to search for allocated data and > holes in the underlying file. > --- > plugins/file/file.c | 139 ++++++++++++++++++++++++++++++++++++++++---- > 1 file changed, 129 insertions(+), 10 deletions(-) > > -int file_debug_zero; /* to enable: -D file.zero=1 */ > +/* Any callbacks

v2 was here: https://www.redhat.com/archives/libguestfs/2020-March/msg00154.html v3: - Micro-optimize tmpdir. - Quote $disk in default command shell fragment. - Don't redirect mkfs output to /dev/null. Instead use exec </dev/null >/dev/null before the shell fragment. We may want to do this in other places where we run external shell scripts, or more generally for all

The NBD protocol is adding an extension to let servers advertise initialization state to the client: whether the image contains holes, and whether it is known to read as all zeroes. For file-based plugins, we are already probing lseek(SEEK_HOLE) to learn if extents are supported; a slight tweak to remember if that result is EOF tells us if we are sparse, and a similar lseek(SEEK_DATA) returning

This plugin now implements efficient zeroing, pre-fetch and extents, which should give quite a performance boost. XXX On the flip side, it no longer ignores flush and FUA (which we can ignore because these are temporary disks), which very likely has a large negative impact on performance. Fixing this would involve generalising fileops a little. --- plugins/tmpdisk/Makefile.am | 2 +

--- common/include/isaligned.h | 11 +++++------ plugins/file/file.c | 4 ++-- plugins/vddk/vddk.c | 8 ++++---- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/common/include/isaligned.h b/common/include/isaligned.h index e693820..81ce8a7 100644 --- a/common/include/isaligned.h +++ b/common/include/isaligned.h @@ -36,16 +36,15 @@ #include <assert.h>

Not sure what version we're up to, but this reimplements extents using the new simpler structure described in this thread: https://www.redhat.com/archives/libguestfs/2019-March/msg00077.html I also fixed most of the things that Eric pointed out in the previous review, although I need to go back over his replies and check I've got everything. This needs a bit more testing. However the

--- pxzcat.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/pxzcat.c b/pxzcat.c index 9bcdc36..55ccfc0 100644 --- a/pxzcat.c +++ b/pxzcat.c @@ -44,10 +44,11 @@ #include <sys/types.h> #include <error.h> #include <errno.h> #include <getopt.h> #include <pthread.h> +#include <linux/falloc.h> #include <lzma.h> #define DEBUG

This can be used for creating temporary disks to thin clients, as a kind of "remote tmpfs". See also: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html --- plugins/data/nbdkit-data-plugin.pod | 1 + plugins/file/nbdkit-file-plugin.pod | 1 + plugins/linuxdisk/nbdkit-linuxdisk-plugin.pod | 7 +- plugins/memory/nbdkit-memory-plugin.pod |