similar to: [RFC PATCH] protocol: Add NBD_CMD_FLAG_FAST_ZERO

While it may be counterintuitive at first, the introduction of NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance regression in qemu [1], when copying a sparse file. When the destination file must contain the same contents as the source, but it is not known in advance whether the destination started life with all zero content, then there are cases where it is faster to request

On Fri, Mar 22, 2019 at 6:43 PM Eric Blake <eblake@redhat.com> wrote: > While it may be counterintuitive at first, the introduction of > NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance > regression in qemu [1], when copying a sparse file. When the > destination file must contain the same contents as the source, but it > is not known in advance whether

On Fri, Mar 22, 2019 at 12:17:59PM -0500, Eric Blake wrote: > On 3/22/19 11:42 AM, Eric Blake wrote: > > > > > Hence, it is desirable to have a way for clients to specify that a > > particular write zero request is being attempted for a fast wipe, and > > get an immediate failure if the zero request would otherwise take the > > same time as a write.

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a

Qemu was able to demonstrate that knowing whether a zero operation is fast is useful when copying from one image to another: there is a choice between bulk pre-zeroing and then revisiting the data sections (fewer transactions, but depends on the zeroing to be fast), vs. visiting every portion of the disk only once (more transactions, but no time lost to duplicated I/O due to slow zeroes). As

On 8/28/19 4:57 AM, Vladimir Sementsov-Ogievskiy wrote: >> Hence, it is desirable to have a way for clients to specify that a >> particular write zero request is being attempted for a fast wipe, and >> get an immediate failure if the zero request would otherwise take the >> same time as a write. Conversely, if the client is not performing a >> pre-initialization

On 3/22/19 11:42 AM, Eric Blake wrote: > > Hence, it is desirable to have a way for clients to specify that a > particular write zero request is being attempted for a fast wipe, and > get an immediate failure if the zero request would otherwise take the > same time as a write. Conversely, if the client is not performing a > pre-initialization pass, it is still more efficient

On 8/23/19 1:48 PM, Wouter Verhelst wrote: > On Fri, Aug 23, 2019 at 09:34:26AM -0500, Eric Blake wrote: >> +- bit 4, `NBD_CMD_FLAG_FAST_ZERO`; valid during >> + `NBD_CMD_WRITE_ZEROES`. If set, but the server cannot perform the >> + write zeroes any faster than it would for an equivalent >> + `NBD_CMD_WRITE`, > > One way of fulfilling the letter of this

Allow plugins to affect the handling of the new NBD_CMD_FLAG_FAST_ZERO flag, then update affected plugins. In particular, in-memory plugins are always fast; the full plugin is better served by omitting .zero and relying on .pwrite fallback; nbd takes advantage of libnbd extensions proposed in parallel to pass through support; and v2 language bindings expose the choice to their scripts. The

v2 was here: https://lists.debian.org/nbd/2022/11/msg00030.html The bulk of the changes since then are: - forbid NBD_OPT_EXPORT_NAME once extended headers are negotiated (Wouter) - consistently use 'maximum payload', rather than a haphazard mix of 'maximum block payload' (Vladimir) At this point, I want to make sure we are happy with the spec before re-posting patches for

Another couple things I noticed that are worth improving, but aren't strictly related to implementing fast zero support. Eric Blake (2): server: Assert sane error responses nozero: More efficient FUA handling filters/nozero/nozero.c | 17 +++++++++++-- server/filters.c | 56 +++++++++++++++++++++++++++++++++-------- server/protocol.c | 32 +++++++++++++++++------ 3 files

Am 10.04.2018 um 15:03 hat Nir Soffer geschrieben: > On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > We now have true zeroing support in oVirt imageio, thanks for that. > > > > However a problem is that ‘qemu-img convert’ issues zero requests for > > the whole disk before starting the transfer. It does this using 32 MB

I ran into these while trying to prepare patches to add NBD_CMD_FLAG_FAST_ZERO, which will expose a new NBD_ENOTSUP wire value. Eric Blake (2): plugins: Don't lose original error when emulating FUA plugins: Permit ENOTSUP as synonym for EOPNOTSUPP docs/nbdkit-filter.pod | 11 ++++++----- docs/nbdkit-plugin.pod | 12 +++++++----- plugins/file/file.c | 16 +++++++++++-----

Qemu was able to demonstrate that knowing whether a zero operation is fast is useful when copying from one image to another: there is a choice between bulk pre-zeroing and then revisiting the data sections (fewer transactions, but depends on the zeroing to be fast), vs. visiting every portion of the disk only once (more transactions, but no time lost to duplicated I/O due to slow zeroes). As

Eventually nbdkit will install this as a public header file and we will be able to consume it from $includedir. In the mean time synchronize with the copy in nbdkit. --- lib/nbd-protocol.h | 57 +++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 29 deletions(-) diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index 699aa22..7df411a 100644 ---

On Tue, Apr 23, 2019 at 07:38:41AM -0500, Eric Blake wrote: > [adding NBD list] > > On 4/23/19 2:36 AM, Richard W.M. Jones wrote: > > On Mon, Apr 22, 2019 at 07:50:22PM -0500, Eric Blake wrote: > >> Previously, we were squashing EOVERFLOW into EINVAL; continue to do so > >> at points in the protocol where the client may not be expecting > >> EOVERFLOW.

On 8/30/19 1:07 PM, Vladimir Sementsov-Ogievskiy wrote: > 23.08.2019 17:37, Eric Blake wrote: >> Commit fe0480d6 and friends added BDRV_REQ_NO_FALLBACK as a way to >> avoid wasting time on a preliminary write-zero request that will later >> be rewritten by actual data, if it is known that the write-zero >> request will use a slow fallback; but in doing so, could not

Similar to the recent --mask-handshake command line added to nbdkit to test client fallbacks to crippled servers, it can be worth testing server fallbacks to crippled clients. And just as we have exposed whether the client will request structured replies, we can also expose whether the client will understand various handshake flags from the NBD protocol. Of course, we default to supporting all

The upstream protocol recently promoted NBD_CMD_WRITE_ZEROES from experimental to a documented extension. Exposing support for this allows plugin writers to create sparse files when driven by a client that knows how to use the extension; meanwhile, even if a plugin does not support this extension, the server benefits from less network traffic from the client. Eric Blake (5): protocol: Support

On Fri, Aug 23, 2019 at 09:34:26AM -0500, Eric Blake wrote: > +- bit 4, `NBD_CMD_FLAG_FAST_ZERO`; valid during > + `NBD_CMD_WRITE_ZEROES`. If set, but the server cannot perform the > + write zeroes any faster than it would for an equivalent > + `NBD_CMD_WRITE`, One way of fulfilling the letter of this requirement but not its spirit could be to have background writes; that is, the