similar to: [nbdkit PATCH] connections: Improve error responses

The upstream protocol recently promoted NBD_CMD_WRITE_ZEROES from experimental to a documented extension. Exposing support for this allows plugin writers to create sparse files when driven by a client that knows how to use the extension; meanwhile, even if a plugin does not support this extension, the server benefits from less network traffic from the client. Eric Blake (5): protocol: Support

Now instead of validating just the client's request, we are validating that all filters are passing valid requests on down. In protocol.c, we were able to assert that our computation of eflags populated all of the flags, and thus calls such as backend_can_write would not fail; however, with filters, keeping those assertions mean the burden is now on the filter to avoid calling into

The code handling the NBD protocol was located in the same file as the code handling connections, for not really any reason except historical. This is quite a large code movement which splits out the protocol code into four new files: server/protocol-handshake.c initial handshake server/protocol-handshake-newstyle.c " " (newstyle)

These are a couple of patches in preparation for the Block Status implementation. While the patches (especially the second one) are very large they are really just elementary code motion. Rich.

This commit implements the core NBD protocol for the "base:allocation" Block Status replies. --- server/internal.h | 7 ++ server/protocol.h | 15 +++ server/protocol-handshake-newstyle.c | 81 ++++++++++++++- server/protocol.c | 141 ++++++++++++++++++++++++--- 4 files changed, 229 insertions(+), 15 deletions(-) diff --git

With some crufty sed scripts we can generate functions that map from NBD protocol flags (eg. NBD_CMD_READ) to strings ("NBD_CMD_READ"). This works on GNU sed and with FreeBSD, also with GNU sed's --posix option, so I guess the sed code is POSIX-compatible. Rich.

This commit implements the core NBD protocol for the "base:allocation" Block Status replies. --- server/internal.h | 7 + server/protocol.h | 17 +- server/protocol-handshake-newstyle.c | 79 ++++++++- server/protocol.c | 248 +++++++++++++++++++++++++-- 4 files changed, 335 insertions(+), 16 deletions(-) diff --git

Similar to the previous patch in caching size, we want to avoid calling into the plugin more than once per connection on any of the flag determination callbacks. The following script demonstrates the speedup, where we avoid repeated calls into a slow can_fua. Pre-patch: $ cat script case "$1" in get_size) echo 1m;; can_fua) sleep 1; echo native;; can_write | can_zero |

While it may be counterintuitive at first, the introduction of NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance regression in qemu [1], when copying a sparse file. When the destination file must contain the same contents as the source, but it is not known in advance whether the destination started life with all zero content, then there are cases where it is faster to request

It's easy to use the sh script to demonstrate that nbdkit is inefficiently calling into .get_size, .can_fua, and friends more than necessary. We've also commented on the list in the past that it would be nice to ensure that when filters call into next_ops, they are not violating constraints (as we've have to fix several bugs in the past where we did not have such checking to protect

Tested via: term1$ qemu-nbd -k $PWD/sock -t -f raw -x foo junk --trace=nbd_\* term2$ ./nbdkit -f -v -e bar nbd socket=$PWD/sock export=foo term3$ qemu-io -t none -f raw nbd://localhost:10809/bar --trace=nbd_\* and checking the traces to see that 'w 0 1' vs. 'w -f 0 1' was able to influence whether the FUA flag showed up at the server in term1. Still to go: figure out how to

While it may be counterintuitive at first, the introduction of NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance regression in qemu [1], when copying a sparse file. When the destination file must contain the same contents as the source, but it is not known in advance whether the destination started life with all zero content, then there are cases where it is faster to request

The plugin interface was impliticly using the last errno value as the source for any error code sent over the wire to the client. This is okay for C, but in other language bindings, it gets rather awkward when you can't guarantee what errno will even be set to by the time the binding glue has finished executing. The solution is to expose an explicit utility function for setting the preferred

Am 10.04.2018 um 15:03 hat Nir Soffer geschrieben: > On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > We now have true zeroing support in oVirt imageio, thanks for that. > > > > However a problem is that ‘qemu-img convert’ issues zero requests for > > the whole disk before starting the transfer. It does this using 32 MB

Until the next few patches expose and implement new callbacks for filters and plugins, our initial implementation for NBD_CMD_CACHE is to just blindly advertise the feature, and call into .pread with an ignored buffer. Note that for bisection reasons, this patch treats any use of a filter as a forced .can_cache of false to bypass the filter's caching; once all affected filters are patched to

On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote: > This makes sense if the device is backed by a block device on oVirt side, > and the NBD support efficient zeroing. But in this case the device is backed > by an empty sparse file on NFS, and oVirt does not support yet efficient > zeroing, we just write zeros manually. > > I think should be handled on virt-v2v plugin

On Fri, Mar 22, 2019 at 6:43 PM Eric Blake <eblake@redhat.com> wrote: > While it may be counterintuitive at first, the introduction of > NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance > regression in qemu [1], when copying a sparse file. When the > destination file must contain the same contents as the source, but it > is not known in advance whether

The NBD protocol states that a 0-length request is undefined; we were inconsistent in that we let it through for read, write, and cache, but blocked it for trim, zero, and block_status. The NBD protocol also has documented rules on handling access beyond EOF, but we are currently wasting traffic to the server when we can give the same answer ourselves. Exposing this as a strictness knob gives

On 3/28/19 11:18 AM, Richard W.M. Jones wrote: > This uses lseek SEEK_DATA/SEEK_HOLE to search for allocated data and > holes in the underlying file. > --- > plugins/file/file.c | 141 ++++++++++++++++++++++++++++++++++--- > tests/Makefile.am | 5 ++ > tests/test-file-extents.sh | 57 +++++++++++++++ > 3 files changed, 193 insertions(+), 10 deletions(-)

Our construction of struct command_in_flight was ad hoc; most parameters were set in command_common(), but extent callbacks were done after the fact, and NBD_CMD_DISC was open-coding things. Furthermore, every caller was triggering nbd_internal_run() for the cmd_issue event; doing that in a central place makes it easier for the next patch to improve that logic without duplicating the fix at each