similar to: [nbdkit] Proposed (new) filter API

Filters can be placed in front of plugins to modify their behaviour. This commit adds the <nbdkit-filter.h> header file, the manual page, the ‘filterdir’ directory (like ‘plugindir’), the ‘filters/’ source directory which will contain the actual filters, the ‘--filters’ parameter, and the filters backend logic. --- Makefile.am | 2 +- TODO | 17 +-

Filters can be placed in front of plugins to modify their behaviour. This commit adds the <nbdkit-filter.h> header file, the manual page, the ‘filterdir’ directory (like ‘plugindir’), the ‘filters/’ source directory which will contain the actual filters, the ‘--filters’ parameter, and the filters backend logic. --- Makefile.am | 2 +- TODO | 17 +-

On 01/16/2018 08:40 AM, Richard W.M. Jones wrote: > Here's my second attempt at a filter API. > > As before, .config and .config_complete can only call the > next->.config or next->.config_complete methods in the chain > respectively. > > The change is with the connected functions (get_size, pread, pwrite, > etc.). Here we pass a struct of next functions

This method can be used for plugins to get control after the server has forked and changed user but before it accepts a connection. This is very late and the only real use for this is for a plugin to create background threads for its own use. --- docs/nbdkit-filter.pod | 20 +++++++++++++++----- docs/nbdkit-plugin.pod | 27 ++++++++++++++++++++++++++- include/nbdkit-filter.h | 2 ++

On Sat, Feb 22, 2020 at 05:11:01AM -0600, Eric Blake wrote: > On 2/22/20 4:37 AM, Richard W.M. Jones wrote: > >Another thing I've been thinking about for some time is splitting > >.config_complete into .config_complete + .get_ready (new name TBD). > >At the moment .config_complete is both the place where we finish > >processing config, and also the last chance we get

This can be used to filter a single partition from a disk image. --- TODO | 2 - configure.ac | 1 + filters/Makefile.am | 3 +- filters/delay/delay.c | 12 +-- filters/offset/nbdkit-offset-filter.pod | 17 ++- filters/offset/offset.c |

If you have a plugin which either creates background threads itself or uses a library that creates background threads, it turns out you cannot create these in .get_ready (or earlier). The reason is that nbdkit forks when either daemonizing itself or using the --run option, and fork cancels all the background threads in the child process (the daemonized or captive nbdkit). The only good solution

This very basic filter allows you to select an offset and range within a plugin, for example: nbdkit --filter=offset file file=foo offset=1M range=100M which serves the byte range [ 1M .. 101M-1 ] from file ‘foo’. --- TODO | 2 - configure.ac | 1 + filters/Makefile.am | 3 +- filters/offset/Makefile.am

I'm about to add an 'exportname' filter, and in the process, I noticed a few shortcomings in our API. Time to fix those before the 1.22 release locks our API in stone. Overloading .list_exports in order to determine a canonical export name at .open time is awkward; the two uses (answering NBD_OPT_LIST for a full list, vs. remapping a client's "" into a canonical name

Eric: Did you want to take this one any further? It might be one that we save for > 1.18: https://www.redhat.com/archives/libguestfs/2020-February/thread.html#00206 Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and

The NBD protocol is adding an extension to let servers advertise initialization state to the client: whether the image contains holes, and whether it is known to read as all zeroes. Most filters just pass through the plugin's result (as it is only checked once when first connecting), but we can do useful things in log (mention the setting), extentlist (answer based on the extents we are

No need for filters to specify their model in two different ways; the callback alone is sufficient. Since all filters are in-tree, we can deal with the API/ABI change made here. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/nbdkit-filter.pod | 48 +++++++++++++++------------------ include/nbdkit-filter.h | 6 ++--- filters/cow/cow.c | 2 --

We documented our intent of only allowing a filter to run with the same version of nbdkit it was compiled against, but up to now, were not actually enforcing that - we had only been insisting on the looser notion of a matching ._api_version, which doesn't help when we've forgotten to bump that macro when making incompatible API/ABI changes (see commit 6934d4c1). However, we can't use

Also implements the --filters parameter. --- docs/nbdkit.pod | 21 +- nbdkit.in | 17 +- src/Makefile.am | 1 + src/filters.c | 606 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/internal.h | 23 ++- src/main.c | 114 +++++++++-- src/plugins.c | 11 +- 7 files changed, 772 insertions(+), 21 deletions(-) diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod index

>From the protocol document: "NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to

I like this change. I think we were overloading the config_complete method before to do two different things (complete configuration; do any allocation/housekeeping necessary before we can start serving). The only questions in my mind are whether we want this before 1.18, and whether the name ("get_ready") is a good one. Rich.

I'm about to add an 'exportname' filter, and in the process, I noticed a few shortcomings in our API. Having .default_export makes it easy to answer NBD_INFO_NAME in response to a client request during NBD_OPT_GO, but answering NBD_INFO_DESCRIPTION is awkward - there's no guarantee the export name was given with a description in .list_exports. Note, however, that while we map

Rebased filters patch. Requires current git master + the locks / thread model fix (https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html) So a few changes here since last time: The "introduce filters" and "implement filters" patches are squashed together. I introduced a concept of .prepare and .finalize. These run before and after the data serving phase

This filter can be used to open tar files. It uses the technique first suggested by Eric Blake here: https://www.redhat.com/archives/libguestfs/2020-July/msg00017.html We suggest that nbdkit-tar-plugin is deprecated in nbdkit 1.26, but it might happen later. --- docs/nbdkit-captive.pod | 4 +- filters/offset/nbdkit-offset-filter.pod | 2 +- filters/tar/nbdkit-tar-filter.pod

On Thu, Aug 27, 2020 at 05:03:45PM -0500, Eric Blake wrote: > When extracting an obvious subset of a larger container (ext2, gzip, > partition, tar, xz), it's fairly easy to add a nice updated > description for what the client is seeing. Not all clients request > the description, but if you are worried about this leaking too much > information, it can be silenced with