similar to: [PATCH v3 0/6] NBD 64-bit extensions (spec only)

Add a new negotiation feature where the client and server agree to use larger packet headers on every packet sent during transmission phase. This has two purposes: first, it makes it possible to perform operations like trim, write zeroes, and block status on more than 2^32 bytes in a single command. For NBD_CMD_READ, replies are still implicitly capped by the maximum block payload limits

On Wed, Feb 22, 2023 at 12:05:44PM +0200, Wouter Verhelst wrote: > On Mon, Nov 14, 2022 at 04:46:54PM -0600, Eric Blake wrote: > > #### Simple reply message > > > > @@ -1232,6 +1235,19 @@ The field has the following format: > > will be faster than a regular write). Clients MUST NOT set the > > `NBD_CMD_FLAG_FAST_ZERO` request flag unless this transmission

I'm hoping to implement .extents for the nbd plugin; this is a prerequisite. I'm not sure about patch 3 - if we like it, I'll squash it to 2, if we don't, I think we are okay just dropping it. I'm also wondering if we have to worry about malicious plugins that don't populate the entire .pread buffer in an effort to get nbdkit to expose portions of the heap; my patch 7 loses

Hi Eric, Busy days, busy times. Sorry about the insane delays here. On Mon, Nov 14, 2022 at 04:46:51PM -0600, Eric Blake wrote: > Commit 9f30fedb improved the spec to allow non-payload requests that > exceed any advertised maximum block size. Take this one step further > by permitting the server to use NBD_EOVERFLOW as a hint to the client > when a request is oversize (while

Updated based on other changes that have happened in the meantime: - rely more on cleanup.h (throughout) - split structured read for easier review (patch 2 and 3 were combined in v1) - rely on nbdkit not leaking a server's partial answer (patch 3) - add tests (patch 5) - other bug fixes I found while testing it - drop EOVERFLOW patch for now; it will be separate once upstream NBD protocol

On Tue, Feb 21, 2023 at 05:21:37PM +0200, Wouter Verhelst wrote: > Hi Eric, > > Busy days, busy times. Sorry about the insane delays here. No problem; I've been tackling other things in the meantime too, so this extension has taken far longer than I planned for more reasons than just slow review time. > > On Mon, Nov 14, 2022 at 04:46:51PM -0600, Eric Blake wrote: > >

This is a cover letter for a set of multi-project patch series all designed to implement 64-bit operations in NBD, and demonstrate interoperability of the new extension between projects. v1 of the project was attempted nearly a year ago: https://lists.nongnu.org/archive/html/qemu-devel/2021-12/msg00453.html Since then, I've addressed a lot of preliminary cleanups in libnbd to make it easier

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros.

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd:

In libnbd, we quickly learned that distinguishing between 'handle' (verb for acting on an object) and 'handle' (noun describing which object to act on) could get confusing; we solved it by renaming the latter to 'cookie'. Copy that approach into the NBD spec, and make it obvious that a cookie is opaque data from the point of view of the server. Makes no difference to

On Tue, May 30, 2023 at 04:06:32PM +0200, Laszlo Ersek wrote: > On 5/25/23 15:00, Eric Blake wrote: > > Support sending 64-bit requests if extended headers were negotiated. > > This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an > > extended NBD_CMD_WRITE; this is such a fundamental part of the > > protocol that for now it is easier to silently ignore

libnbd-0.1.4-1 is now available in Fedora 29/30 updates testing. Diffs since v2 - rebase to master, bump from libnbd 0.1.2 to 0.1.3+, add tests to TLS usage which flushed out the need to turn relative pathnames into absolute, doc tweaks Now that the testsuite covers TLS and libnbd has been fixed to provide the things I found lacking when developing v2, I'm leaning towards pushing this on

This is about the simplest implementation of NBD Structured Replies (SR) that we can do right now. It accepts NBD_OPT_STRUCTURED_REPLIES when negotiated by the client, but only sends back the simplest possible SR when required to by NBD_CMD_READ. The rest of the time it will send back simple replies as before. We do not modify the plugin API so plugins are unable to send complex SRs. Also we

I've mentioned this topic before (in fact, the idea of adding NBD_CMD_FLAG_DF was first mentioned at [1]), but finally finished enough of an implementation to feel confident in posting it. I'd still like to add something under examples/ that uses the new API to implement strict checking of a server's structured replies read implementation (ensure that a server never sends data after

On Mon, Jun 17, 2019 at 07:07:53PM -0500, Eric Blake wrote: > The NBD spec requires that a server doing structured reads must not > succeed unless it covers the entire buffer with reply chunks. In the > general case, this requires a lot of bookkeeping to check whether > offsets were non-overlapping and sufficient, and we'd rather defer > such checking to an optional callback

None are supported at present, so this always returns an empty list. --- docs/nbdkit-protocol.pod | 4 ++ server/protocol.h | 2 + server/connections.c | 84 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 90 insertions(+) diff --git a/docs/nbdkit-protocol.pod b/docs/nbdkit-protocol.pod index 68438fa..4c65e00 100644 --- a/docs/nbdkit-protocol.pod +++

On Wed, Feb 22, 2023 at 11:49:18AM +0200, Wouter Verhelst wrote: > On Mon, Nov 14, 2022 at 04:46:52PM -0600, Eric Blake wrote: > [...] > > @@ -1370,9 +1475,10 @@ of the newstyle negotiation. > > Return a list of `NBD_REP_META_CONTEXT` replies, one per context, > > followed by an `NBD_REP_ACK` or an error. > > > > - This option SHOULD NOT be

nbdkit (upstream 5a7a394c699) currently fails with qemu 2.12.0: $ ./nbdkit memory size=64M --run 'qemu-img convert $nbd /var/tmp/out' nbdkit: memory.2: error: invalid request: unknown command (7) ignored qemu-img: Protocol error: simple reply when structured reply chunk was expected This was a bug in qemu which was fixed upstream quite a long time ago by the commit I've

Patches 1 & 2 are rather complex, but the end result is that we pass closures + user_data + free function in single struct parameters as I described previously in this email: https://www.redhat.com/archives/libguestfs/2019-August/msg00210.html Patch 3 adds a convenient FREE_CALLBACK macro which seems a worthwhile simplification if you buy into 1 & 2. Patch 4 adds another macro which is

This was v3 patch 2/22, reworked to address the confusion about how a structured reply header is read in two pieces before getting to the payload portion. I'm still working on rebasing the rest of my v3 series (patches 1, 3-22) from other comments given, but this seemed independent enough that it's worth posting now rather than holding it up for the rest of the series. Eric Blake (4):