similar to: [nbdkit PATCH 0/7] Initial implementation of FUA flag passthrough

A combination of the work that both Rich and I have been doing lately, where filters use only the new API with flags on every command that the client can send over the wire (we can then add support for more flags in nbdkit without having to add new callbacks, as NBD adds more flags upstream). Eric Blake (4): protocol: Split flags from cmd field in requests backend: Pass flags argument through

After more than a month since v2 [1], I've finally got my FUA support series polished. This is all of my outstanding patches, even though some of them were originally posted in separate threads from the original FUA post [2], [3] [1] https://www.redhat.com/archives/libguestfs/2018-January/msg00113.html [2] https://www.redhat.com/archives/libguestfs/2018-January/msg00219.html [3]

On 01/16/2018 02:25 AM, Richard W.M. Jones wrote: > I had an idea about how we might do this and still keep a single > .pwrite method for plugins. Maybe it's too complicated but here goes: > > > This is binary-compatible with old plugins, but not source compatible, so: > > > (2) Plugsin may optionally define NBDKIT_PLUGIN_LEVEL before including >

Hi, We've been using a modified nbdkit (ours is cbdkit internally) for about half a year now and since you guys appear to be working on a next version of the API I wanted to go over some of the limitations we hit with nbdkit that we think others may also hit for consideration into the batch of changes you are making to the api. About Us: Our primary use for nbdkit is to facilitate a disk

[still a work in progress, as I finish rebasing to capture the ideas raised on the list, but posting now for initial feedback] The NBD protocol supports Forced Unit Access (FUA) as a more efficient way to wait for just one write to land in persistent storage, rather than all outstanding writes at the time of a flush; modeled after the kernel's block I/O flag of the same name. While we can

Here's what I'm currently playing with; I'm not ready to commit anything until I rebase my FUA work on top of this, as I only want to break filter ABI once between releases. Eric Blake (2): backend: Rework internal/filter error return semantics filters: Add log filter TODO | 2 - docs/nbdkit-filter.pod | 84 +++++++-- docs/nbdkit.pod

It's easy to use the sh script to demonstrate that nbdkit is inefficiently calling into .get_size, .can_fua, and friends more than necessary. We've also commented on the list in the past that it would be nice to ensure that when filters call into next_ops, they are not violating constraints (as we've have to fix several bugs in the past where we did not have such checking to protect

Since v1: add the blocksize filter, add testsuite coverage of the log filter, several fixes to the log filter based on what adding tests revealed I'm still working on FUA flag support patches on top of this; the patches should all be committed in the same release, as we want to minimize the number of releases that cause a filter ABI/API bump Eric Blake (3): backend: Rework internal/filter

v1 -> v2: - Fixed everything mentioned in the review. Rich.

Somewhat invasive but mostly mechanical change to how plugins are called. This patch is in preparation for adding a second backend subtype for filters. Rich.

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that

I still don't have .prepare/.finalize working cleanly across reopen, but did find a nasty bug where a botched assertion means we failed to notice reads beyond EOF in both the xz and retry filter. Refactoring backend.c will make .finalize work easier. Eric Blake (5): xz: Avoid reading beyond EOF retry: Check size before transactions tests: Test retry when get_size values change

The first three patches are identical to: https://www.redhat.com/archives/libguestfs/2018-January/msg00079.html "[PATCH nbdkit v2 0/3] Refactor plugin_* functions into a backend" The rest of the patches add filters using the new filter API previously described here: https://www.redhat.com/archives/libguestfs/2018-January/msg00073.html This needs a lot more testing -- and tests --

On 01/20/2017 02:16 PM, Eric Blake wrote: > Reject rather than silently ignoring unknown client request flags. > > > + /* Validate flags */ > + if (flags & ~NBD_CMD_FLAG_FUA) { > + nbdkit_error ("invalid request: unknown flag (0x%x)", flags); > + *error = EINVAL; > + return 0; > + } Right now, our NBD_CMD_FLAG_FUA implementation causes a

For existing commits, this is almost identical to v1, except that I updated some commit messages and reordered the commits in a somewhat more logical sequence. The main changes are the extra commits: [06/11] plugins: Return NBD_FLAG_CAN_MULTI_CONN from some readonly plugins. - Readonly plugins that can set the flag unconditionally. [09/11] partitioning: Return NBD_FLAG_CAN_MULTI_CONN. [10/11]

I ran some local testing on my Fedora 29 system via: $ ./nbdkit memory 1m $ for i in `seq 10`; do ./nbdkit -U - --filter=stats nbd \ statsappend=true statsfile=$file hostname=localhost port=10809 \ --run '~/libnbd/examples/threaded-reads-and-writes $unixsocket' done Pre-patch, the runs averaged 1.266s, 1.30E+08 bits/s Post-patch, the runs averaged 1.154s, 1.42E+08

On Fri, Jan 19, 2018 at 07:40:29AM -0600, Eric Blake wrote: > =head1 SYNOPSIS > > + #define NBDKIT_API_VERSION 2 > + > #include <nbdkit-plugin.h> > > #define THREAD_MODEL NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS > @@ -51,9 +53,21 @@ L<nbdkit-perl-plugin(3)>, > L<nbdkit-python-plugin(3)>, > L<nbdkit-ruby-plugin(3)>. > >

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a

The upstream protocol recently promoted NBD_CMD_WRITE_ZEROES from experimental to a documented extension. Exposing support for this allows plugin writers to create sparse files when driven by a client that knows how to use the extension; meanwhile, even if a plugin does not support this extension, the server benefits from less network traffic from the client. Eric Blake (5): protocol: Support

On 01/26/2017 04:18 AM, Richard W.M. Jones wrote: > On Wed, Jan 25, 2017 at 08:55:18PM -0600, Eric Blake wrote: >> On 01/20/2017 02:16 PM, Eric Blake wrote: >>> Reject rather than silently ignoring unknown client request flags. >>> >> >>> >>> + /* Validate flags */ >>> + if (flags & ~NBD_CMD_FLAG_FUA) { >>> +