similar to: [nbdkit PATCH] plugins: Use static buffer for plugin_zeroes

The NBD protocol supports Forced Unit Access (FUA) as a more efficient way to wait for just one write to land in persistent storage, rather than all outstanding writes at the time of a flush; modeled after the kernel's block I/O flag of the same name. While we can emulate the proper semantics with a full-blown flush, there are some plugins that can properly pass the FUA flag on to the end

Introduce the concept of a backend. Currently the only type of backend is a plugin, and there can only be one of them. Instead of calling functions like ‘plugin_pwrite’ you call the backend method ‘backend->pwrite (backend, ...)’. The change is largely mechanical. I was able to remove ‘assert (dl)’ statements throughout since we can now prove they will never be called. Note this does not

Introduce the concept of a backend. Currently the only type of backend is a plugin, and there can only be one of them. Instead of calling functions like ‘plugin_pwrite’ you call the backend method ‘backend->pwrite (backend, ...)’. The change is largely mechanical. I was able to remove ‘assert (dl)’ statements throughout since we can now prove they will never be called. Note this does not

Introduce the concept of a backend. Currently the only type of backend is a plugin, and there can only be one of them. Instead of calling functions like ‘plugin_pwrite’ you call the backend method ‘backend->pwrite (backend, ...)’. The change is largely mechanical. I was able to remove ‘assert (dl)’ statements throughout since we can now prove they will never be called. Note this does not

This is a quick hack to experiment with parallel threading model in the python plugin. Changes: - Use aligned buffers to make it possible to use O_DIRECT. Using parallel I/O does not buy us much when using buffered I/O. pwrite() copies data to the page cache, and pread() reads data from the page cache. - Disable extents in the file plugin. This way we can compare it with the python

Somewhat invasive but mostly mechanical change to how plugins are called. This patch is in preparation for adding a second backend subtype for filters. Rich.

Previously, we let a plugin set an error in either thread-local storage (nbdkit_set_error()) or errno, then connections.c would decode which error to use. But with filters in the mix, it is very difficult for a filter to know what error was set by the plugin (particularly since nbdkit_set_error() has no public counterpart for reading the thread-local storage). What's more, if a filter does

v1 -> v2: - Fixed everything mentioned in the review. Rich.

From: "Richard W.M. Jones" <rjones@redhat.com> Previously only one handle could be stored, but we will need to store multiple handles when we have filters. The plugin handle is defined as index 0. Filters will use indices > 0. Message-Id: <20180117205356.8699-6-rjones@redhat.com> [eblake: rework for FUA support] Signed-off-by: Eric Blake <eblake@redhat.com> ---

This is a quick hack to experiment with parallel threading model in the python plugin. Changes: - Use aligned buffers to make it possible to use O_DIRECT. Using parallel I/O does not buy us much when using buffered I/O. pwrite() copies data to the page cache, and pread() reads data from the page cache. - Disable extents in the file plugin. This way we can compare it with the python

On Thu, Aug 06, 2020 at 11:22:00PM +0300, Nir Soffer wrote: > This is a quick hack to experiment with parallel threading model in the > python plugin. > > Changes: > > - Use aligned buffers to make it possible to use O_DIRECT. Using > parallel I/O does not buy us much when using buffered I/O. pwrite() > copies data to the page cache, and pread() reads data from the

I still don't have .prepare/.finalize working cleanly across reopen, but did find a nasty bug where a botched assertion means we failed to notice reads beyond EOF in both the xz and retry filter. Refactoring backend.c will make .finalize work easier. Eric Blake (5): xz: Avoid reading beyond EOF retry: Check size before transactions tests: Test retry when get_size values change

Previously only one handle could be stored, but we will need to store multiple handles when we have filters. The plugin handle is defined as index 0. Filters will use indices > 0. --- src/connections.c | 37 ++++++++++++++++++++++++++++++------- src/internal.h | 4 ++-- src/plugins.c | 53 +++++++++++++++++++++++++++-------------------------- 3 files changed, 59 insertions(+), 35

The plugin interface was impliticly using the last errno value as the source for any error code sent over the wire to the client. This is okay for C, but in other language bindings, it gets rather awkward when you can't guarantee what errno will even be set to by the time the binding glue has finished executing. The solution is to expose an explicit utility function for setting the preferred

A combination of the work that both Rich and I have been doing lately, where filters use only the new API with flags on every command that the client can send over the wire (we can then add support for more flags in nbdkit without having to add new callbacks, as NBD adds more flags upstream). Eric Blake (4): protocol: Split flags from cmd field in requests backend: Pass flags argument through

Until the next few patches expose and implement new callbacks for filters and plugins, our initial implementation for NBD_CMD_CACHE is to just blindly advertise the feature, and call into .pread with an ignored buffer. Note that for bisection reasons, this patch treats any use of a filter as a forced .can_cache of false to bypass the filter's caching; once all affected filters are patched to

The first three patches are identical to: https://www.redhat.com/archives/libguestfs/2018-January/msg00079.html "[PATCH nbdkit v2 0/3] Refactor plugin_* functions into a backend" The rest of the patches add filters using the new filter API previously described here: https://www.redhat.com/archives/libguestfs/2018-January/msg00073.html This needs a lot more testing -- and tests --

The upstream protocol recently promoted NBD_CMD_WRITE_ZEROES from experimental to a documented extension. Exposing support for this allows plugin writers to create sparse files when driven by a client that knows how to use the extension; meanwhile, even if a plugin does not support this extension, the server benefits from less network traffic from the client. Eric Blake (5): protocol: Support

Tested via: term1$ qemu-nbd -k $PWD/sock -t -f raw -x foo junk --trace=nbd_\* term2$ ./nbdkit -f -v -e bar nbd socket=$PWD/sock export=foo term3$ qemu-io -t none -f raw nbd://localhost:10809/bar --trace=nbd_\* and checking the traces to see that 'w 0 1' vs. 'w -f 0 1' was able to influence whether the FUA flag showed up at the server in term1. Still to go: figure out how to

Here's what I'm currently playing with; I'm not ready to commit anything until I rebase my FUA work on top of this, as I only want to break filter ABI once between releases. Eric Blake (2): backend: Rework internal/filter error return semantics filters: Add log filter TODO | 2 - docs/nbdkit-filter.pod | 84 +++++++-- docs/nbdkit.pod