similar to: [PATCH nbdkit] New filter: gzip

On 7/10/20 6:50 AM, Richard W.M. Jones wrote: > Turn the existing nbdkit-gzip-plugin into a filter so it can be > applied on top of files or other sources: > > nbdkit file --filter=gzip file.gz > nbdkit curl --filter=gzip https://example.com/disk.gz > > Because of the nature of the gzip format which is not blocked based > and thus not seekable, this filter caches

On Thu, Aug 27, 2020 at 05:03:45PM -0500, Eric Blake wrote: > When extracting an obvious subset of a larger container (ext2, gzip, > partition, tar, xz), it's fairly easy to add a nice updated > description for what the client is seeing. Not all clients request > the description, but if you are worried about this leaking too much > information, it can be silenced with

When extracting an obvious subset of a larger container (ext2, gzip, partition, tar, xz), it's fairly easy to add a nice updated description for what the client is seeing. Not all clients request the description, but if you are worried about this leaking too much information, it can be silenced with nbdkit-exportname-filter. Signed-off-by: Eric Blake <eblake@redhat.com> --- And maybe

The NBD protocol is adding an extension to let servers advertise initialization state to the client: whether the image contains holes, and whether it is known to read as all zeroes. Most filters just pass through the plugin's result (as it is only checked once when first connecting), but we can do useful things in log (mention the setting), extentlist (answer based on the extents we are

From: "Richard W.M. Jones" <rjones@redhat.com> This patch may be worth squashing? --- docs/nbdkit-filter.pod | 65 +++++++++++++++++++++++++++++++++++++++++++------ include/nbdkit-filter.h | 29 ++++++++++++---------- src/filters.c | 55 ++++++++++++++++++++--------------------- 3 files changed, 101 insertions(+), 48 deletions(-) diff --git a/docs/nbdkit-filter.pod

When querying the extents of the underlying plugin, we should only translate the starting offset, and let the plugin report for at least as many bytes as our range permits. Otherwise, short-changing the range causes bad behavior such as returning 0 extents, or even failing the creation of an extents tracker: $ cat script case "$1" in get_size) echo 1m;; can_extents) ;;

Although the time spent in memcpy/memset probably pales in comparison to time spent in socket I/O, it's still worth worth reducing the number of times we have to utilize a bounce buffer when we already have aligned data. Signed-off-by: Eric Blake <eblake@redhat.com> --- filters/cache/cache.c | 60 ++++++++++++++++++++++++++++--------------- 1 file changed, 39 insertions(+), 21

On 7/22/20 7:42 AM, Richard W.M. Jones wrote: > See also: > https://www.redhat.com/archives/libguestfs/2020-July/msg00090.html > --- > docs/nbdkit-plugin.pod | 57 +++++++++++++++++++++++++++++++++++++--- > docs/nbdkit-protocol.pod | 7 +++-- > 2 files changed, 58 insertions(+), 6 deletions(-) I'm playing with code for an initial implementation of this, and quickly

On Thu, Oct 03, 2019 at 09:54:37PM -0500, Eric Blake wrote: > Although it is unusual for a plugin to shrink size on reopen, it is > not impossible. Failure to check our bounds could result in violating > assumptions in the plugin that all requests are in-bounds. Note that > if the plugin gains a larger size, we merely never access the new tail > of the file (when the NBD protocol

It may be useful to test whether the client's use of NBD_CMD_FLAG_NO_HOLE makes a difference; do this by adding a mode to --filter=nozero to force a non-trimming zero write. Signed-off-by: Eric Blake <eblake@redhat.com> --- filters/nozero/nbdkit-nozero-filter.pod | 19 +++++++---- filters/nozero/nozero.c | 45 ++++++++++++++++++++----- tests/test-nozero.sh

I'm impressed that I was able to whip this out in just one day of hacking. Below, I'll include a diff between the plugin and the filter as of patch 1, if it aids review. Eric Blake (3): filters: Add ext2 filter ext2: Deprecate ext2 plugin ext2: Add mode for letting client exportname choose file from image TODO | 5 - configure.ac

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_on=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map` runs

Applies on top of my pending series for the exportname filter, addressing one of the todo's in that cover letter. Eric Blake (2): filters: Add .export_description wrappers ext2: Supply .list_exports and .default_export filters/ext2/nbdkit-ext2-filter.pod | 3 +- tests/Makefile.am | 16 +++- filters/ext2/ext2.c | 125 +++++++++++++++++++---------

With our recent cleanups to nxdata, the only remaining difference between functions like backend_open() and next_open() was the signature (one used void*, the other struct backend *); the API is compatible. All of our filters are in-tree, and we don't promise API/ABI stability, but it is still a lot of files to touch, so the simplest solution to avoid the redundant hop through wrapper

This can be used to filter a single partition from a disk image. --- TODO | 2 - configure.ac | 1 + filters/Makefile.am | 3 +- filters/delay/delay.c | 12 +-- filters/offset/nbdkit-offset-filter.pod | 17 ++- filters/offset/offset.c |

Take advantage of the fact that we can now detect the type of client during --tls=on in order to provide safe dummy content for plaintext clients without having to rewrite plugins to do so. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/nbdkit-plugin.pod | 4 +- docs/nbdkit-tls.pod | 5 +- filters/tlsdummy/nbdkit-tlsdummy-filter.pod

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_on=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map` runs

On 01/19/2018 09:23 AM, Richard W.M. Jones wrote: > Filters can be placed in front of plugins to modify their behaviour. > > This commit adds the <nbdkit-filter.h> header file, the manual page, > the ‘filterdir’ directory (like ‘plugindir’), the ‘filters/’ source > directory which will contain the actual filters, the ‘--filters’ > parameter, and the filters backend logic.

https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76 Already existing filters (readahead, cache) could be improved if filters could open a background work thread or threads which connect independently to the plugin. A proposed new filter (scan) cannot really be written at all without this capability. First of all the reason this can't be done today is

>From the protocol document: "NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to