search for: fast_zero

...ime lost to duplicated I/O due to slow zeroes). As such, the NBD protocol is adding an extension to allow clients to request fast failure when zero is not efficient, from servers that advertise support for the new flag. In libnbd, this results in the addition of a new flag, a new functoin nbd_can_fast_zero, and the ability to use the new flag in nbd_zero variants. It also enhances the testsuite to test the flag against new enough nbdkit, which is being patched at the same time. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/nbd-protocol.h | 2 ++ generator/generator | 29 +++++...

Add coverage for the previous patch. Although most plugins don't change can_fua or can_fast_zero on the fly, the code should still not abort when calling .zero with a flag that was permitted by the previous open but not permitted by the current open. Signed-off-by: Eric Blake <eblake@redhat.com> --- tests/Makefile.am | 2 + tests/test-retry-zero-flags.sh | 126 ++++++++++...

...only once (more transactions, but no time lost to duplicated I/O due to slow zeroes). As such, the NBD protocol is adding an extension to allow clients to request fast failure when zero is not efficient, from servers that advertise support for the new flag. In nbdkit, start by plumbing a new .can_fast_zero through the backends (although it stays 0 in this patch, later patches will actually expose it to plugins and filters to override). Advertise the flag to the client when the plugin provides a .can_fast_zero, and wire in passing the flag down to .zero in the plugin. In turn, if the flag is set and...

...chives/libguestfs/2019-August/msg00322.html Based-on: https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg04805.html [Andrey Shinkevich block: workaround for unaligned byte range in fallocate()] Eric Blake (5): nbd: Improve per-export flag handling in server nbd: Prepare for NBD_CMD_FLAG_FAST_ZERO nbd: Implement client use of NBD FAST_ZERO nbd: Implement server use of NBD FAST_ZERO nbd: Tolerate more errors to structured reply request docs/interop/nbd.txt | 3 +- include/block/nbd.h | 6 +++- block/nbd.c | 7 +++++ blockdev-nbd.c | 3 +- nbd/client.c | 39...

...r without * modification, are permitted provided that the following conditions are @@ -262,14 +262,17 @@ log_prepare (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle, int e = next_ops->can_extents (nxdata); int c = next_ops->can_cache (nxdata); int Z = next_ops->can_fast_zero (nxdata); + int is = next_ops->init_sparse (nxdata); + int iz = next_ops->init_zero (nxdata); if (size < 0 || w < 0 || f < 0 || r < 0 || t < 0 || z < 0 || F < 0 || - e < 0 || c < 0 || Z < 0) + e < 0 || c < 0 || Z < 0 || is < 0 || iz &l...

I'm still working on another set of patches to have reopen call .finalize/.prepare (so that another filter can safely appear between retry and the plugin), but for tonight, these are the patches I think are ready to go. Eric Blake (4): retry: Handle can_fua and can_fast_zero changes tests: Test retry with different fua/fast-zero flags server: Close backends if a filter's .open fails server: Better documentation of .open ordering docs/nbdkit-filter.pod | 17 +++-- server/backend.c | 5 +- server/connections.c | 2 +- serv...

...020-08-06 02:07:23.080415 ListExports id=1 readonly=0 default_only=0 ... 2020-08-06 02:07:23.080502 ...ListExports id=1 exports=[""] return=0 - 2020-08-06 02:07:23.080712 connection=1 Connect export='' size=0x400 write=1 flush=1 rotational=0 trim=1 zero=2 fua=2 extents=1 cache=2 fast_zero=1 + 2020-08-06 02:07:23.080712 connection=1 Connect export='' tls=0 size=0x400 write=1 flush=1 rotational=0 trim=1 zero=2 fua=2 extents=1 cache=2 fast_zero=1 2020-08-06 02:07:23.080907 connection=1 Read id=1 offset=0x0 count=0x200 ... 2020-08-06 02:07:23.080927 connection=1 ...Read id=1...

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a 'fast-zero' branch to each of: https://rep...

See the cross-post cover letter for details: https://www.redhat.com/archives/libguestfs/2019-August/msg00322.html Eric Blake (1): api: Add support for FAST_ZERO flag lib/nbd-protocol.h | 2 ++ generator/generator | 29 +++++++++++++++++++++++------ lib/flags.c | 12 ++++++++++++ lib/protocol.c | 1 + lib/rw.c | 9 ++++++++- tests/Makefile.am | 20 ++++++++++++++++++++ tests/eflags-plugin.sh | 5 +++++ 7 fi...

Allow plugins to affect the handling of the new NBD_CMD_FLAG_FAST_ZERO flag, then update affected plugins. In particular, in-memory plugins are always fast; the full plugin is better served by omitting .zero and relying on .pwrite fallback; nbd takes advantage of libnbd extensions proposed in parallel to pass through support; and v2 language bindings expose the choic...

..."zero", { default_call with args = [ UInt64 "count"; UInt64 "offset" ]; - optargs = [ OFlags ("flags", cmd_flags) ]; + optargs = [ OFlags ("flags", cmd_flags, + Some ["FUA"; "NO_HOLE"; "FAST_ZERO"]) ]; ret = RErr; permitted_states = [ Connected ]; shortdesc = "send write zeroes command to the NBD server"; @@ -1733,7 +1739,7 @@ cannot do this, see L<nbd_can_fast_zero(3)>)."; "block_status", { default_call with args = [ UInt64 &q...

...(1<<2) #define NBD_CMD_FLAG_REQ_ONE (1<<3) +*/ /* NBD error codes. */ #define NBD_SUCCESS 0 diff --git a/generator/generator b/generator/generator index a72f36c..170121e 100755 --- a/generator/generator +++ b/generator/generator @@ -971,7 +971,14 @@ let cmd_flags = { "FAST_ZERO", 1 lsl 4; ] } -let all_flags = [ cmd_flags ] +let handshake_flags = { + flag_prefix = "HANDSHAKE_FLAG"; + flags = [ + "FIXED_NEWSTYLE", 1 lsl 0; + "NO_ZEROES", 1 lsl 1; + ] +} +let all_flags = [ cmd_flags; handshake_flags ] (* Calls. * @@...

...o the slower write, than it is for the client to have to perform the fallback to send NBD_CMD_WRITE with a zeroed buffer. Add a protocol flag and corresponding transmission advertisement flag to make it easier for clients to inform the server of their intent. If the server advertises NBD_FLAG_SEND_FAST_ZERO, then it promises two things: to perform a fallback to write when the client does not request NBD_CMD_FLAG_FAST_ZERO (so that the client benefits from the lower network overhead); and to fail quickly with ENOTSUP if the client requested the flag but the server cannot write zeroes more efficiently t...

...raise RuntimeError("short write") + +def flush(h, flags): + os.fsync(h['fd']) diff --git a/server/plugins.c b/server/plugins.c index d4364cd2..ce4700a3 100644 --- a/server/plugins.c +++ b/server/plugins.c @@ -631,6 +631,8 @@ plugin_zero (struct backend *b, void *handle, bool fast_zero = flags & NBDKIT_FLAG_FAST_ZERO; bool emulate = false; bool need_flush = false; + void *zero_buffer = NULL; + int buffer_size = MIN (MAX_REQUEST_SIZE, count); if (fua && backend_can_fua (b) != NBDKIT_FUA_NATIVE) { flags &= ~NBDKIT_FLAG_FUA; @@ -669,19 +671,25 @@ pl...

Similar to what I just did for OCaml (this IS an API break, requiring recompilation of any existing Rust plugin), and done because I want to add fast_zero support to both languages as part of my upcoming fast zero series. Figuring out how to get extents working was hard enough that I punted that, still. Eric Blake (2): rust: Implement can_cache rust: Add support for dynamic .thread_model plugins/rust/nbdkit-rust-plugin.pod | 29 ++++++++++++++...

...n, tests and fixes here: https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html remote code execution vulnerability See the full announcement here: https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html New APIs nbd_can_fast_zero(3) Test support by the server for fast zeroing (Eric Blake). nbd_connect_socket(3) nbd_aio_connect_socket(3) Connect to a local connected socket which you create in your main program using your own chosen method. nbd_connect_systemd_socket_act...

Patch 3 still needs tests added, but it is at least working from my simple command line tests. Eric Blake (3): server: Implement nbdkit_is_tls for use during .open server: Expose final thread_model to filter's .get_ready tlsdummy: New filter docs/nbdkit-filter.pod | 21 +- docs/nbdkit-plugin.pod | 34 ++- docs/nbdkit-tls.pod

...raise RuntimeError("short write") + +def flush(h, flags): + os.fsync(h['fd']) diff --git a/server/plugins.c b/server/plugins.c index d4364cd2..ce4700a3 100644 --- a/server/plugins.c +++ b/server/plugins.c @@ -631,6 +631,8 @@ plugin_zero (struct backend *b, void *handle, bool fast_zero = flags & NBDKIT_FLAG_FAST_ZERO; bool emulate = false; bool need_flush = false; + void *zero_buffer = NULL; + int buffer_size = MIN (MAX_REQUEST_SIZE, count); if (fua && backend_can_fua (b) != NBDKIT_FUA_NATIVE) { flags &= ~NBDKIT_FLAG_FUA; @@ -669,19 +671,25 @@ pl...

...s for the > client to have to perform the fallback to send NBD_CMD_WRITE with a > zeroed buffer. > > Add a protocol flag and corresponding transmission advertisement flag > to make it easier for clients to inform the server of their intent. If > the server advertises NBD_FLAG_SEND_FAST_ZERO, then it promises two > things: to perform a fallback to write when the client does not > request NBD_CMD_FLAG_FAST_ZERO (so that the client benefits from the > lower network overhead); and to fail quickly with ENOTSUP if the > client requested the flag but the server cannot write zeroe...

...gt; + > +def flush(h, flags): > + os.fsync(h['fd']) > diff --git a/server/plugins.c b/server/plugins.c > index d4364cd2..ce4700a3 100644 > --- a/server/plugins.c > +++ b/server/plugins.c > @@ -631,6 +631,8 @@ plugin_zero (struct backend *b, void *handle, > bool fast_zero = flags & NBDKIT_FLAG_FAST_ZERO; > bool emulate = false; > bool need_flush = false; > + void *zero_buffer = NULL; > + int buffer_size = MIN (MAX_REQUEST_SIZE, count); > > if (fua && backend_can_fua (b) != NBDKIT_FUA_NATIVE) { > flags &= ~NBDKIT...