similar to: [cross-project PATCH v2] NBD 64-bit extensions

This is a subset of my v2 posting, but limited to just the NBD_OPT_LIST handling. The biggest change since v2 is the addition of added unit testing in all four language bindings (C, python, ocaml, golang). The tests require nbdkit built from git on PATH, and may not be entirely idiomatic, but I at least validated that they catch issues (for example, adding an exit statement near the end of the

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit

An incremental improvement on top of listing exports. I still think it's worth experimenting with revisiting how our API for list mode should actually work [1] (so that we can reuse a single connection for both grabbing the list and finally using NBD_OPT_GO), but this change was easier to whip together while still thinking about that. [1]

I will be following up to this email with four separate threads each addressed to the appropriate single list, with proposed changes to: - the NBD protocol - qemu: both server and client - libnbd: client - nbdkit: server The feature in question adds a new optional NBD_INFO_ packet to the NBD_OPT_GO portion of handshake, adding up to 16 bits of information that the server can advertise to the

In v2: ack'ed preliminary patches have been pushed, and I've added a lot of testsuite coverage as well as putting the new API to use in nbdinfo. Eric Blake (2): api: Add nbd_opt_list_meta_context info: List available meta-contexts lib/internal.h | 1 + generator/API.ml | 84 ++++++++-

RFC because this is an API break, but we haven't declared stable API yet. If we like it, I'm working on using libnbd to implement the nbdkit-nbd plugin; knowing whether it is API version 0.1 or 0.2 will be useful. I also dabbled with allowing optional parameters in python, although my OCaml is weak enough that there may be cleaner ways to approach that. Eric Blake (4): api: Add flags

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):

Rather than moving the connection to DEAD, we can just ignore further contexts to the existing command handle, and fail the overall command with the same errno as the failed callback. Eric Blake (2): states: Track cmd->error as errno, not wire value api: Recover from block status callback failure generator/generator | 5 ++- generator/states-reply-simple.c | 2 +-

Since v1: - rebase to applied patches - split out support for Int in callbacks - sort of test that callbacks work in OCaml (see comment in patch 5) - rename API to nbd_pread_structured - expose error as explicit parameter to callback Eric Blake (5): generator: Allow Int in callbacks states: Wire in a read callback states: Add nbd_pread_structured API states: Add tests for

In v2: - now based on my proposal to add LIBNBD_SHUTDOWN_IMMEDIATE - four flags instead of two: STRICT_FLAGS is new (patch 4), and STRICT_BOUNDS is separate from STRICT_ZERO_SIZE (patch 5) - various refactorings for more shared code and less duplication Eric Blake (5): api: Add xxx_MASK constant for each Flags type generator: Refactor filtering of accepted OFlags api: Add

These bindings get as far as running very simple connections. However there are many missing parts still: * No callbacks. * No functions which handle buffers (pread/pwrite!) This is posted just for general early interest, not even for review. Rich.

On 9/29/20 6:23 AM, Richard W.M. Jones wrote: > This is not related to the current patch, but generally for libnbd: > > (1) Should we be testing interop separately for qemu-storage-daemon > (qsd), or is qsd basically qemu-nbd in a new wrapper so it's not worth > doing it? > > (2) I found a bug in the new nbdinfo behaviour: > > $ nbdkit -fv file dir=/scratch >

This feature is roughly finished now, although it needs a few more tests and some examples. It's pretty much up to par with all the other bindings, but it lacks a completely safe AIO buffer. It won't stop you from freeing the buffer too early) because golang's GC inexplicably lacks a way to declare a root from C. I can probably do it with a global variable and ref counting on the

Now runs a complete set of tests, notably including the AIO test. File descriptors are passed in and out as plain ints (instead of *os.File) for a couple of reasons: (1) We have to pass the plain int to syscall.Select. (2) Turning an fd into an os.File causes golang to set the blocking flag which is deeply unhelpful. Rich.

On Thu, Jul 11, 2019 at 09:35:02AM -0500, Eric Blake wrote: >On 7/11/19 9:23 AM, Martin Kletzander wrote: >> The callback (e.g. for `nbd_block_status`) now has a support for returning >> errors thanks to the last parameter (`int *error`), so it was switched to >> returning void. > >No, the callback still returns int. Where are you seeing it return >void, because

On 9/29/20 9:37 AM, Richard W.M. Jones wrote: > On Tue, Sep 29, 2020 at 06:51:06AM -0500, Eric Blake wrote: >> On 9/29/20 6:23 AM, Richard W.M. Jones wrote: >>> This is not related to the current patch, but generally for libnbd: >>> >>> (1) Should we be testing interop separately for qemu-storage-daemon >>> (qsd), or is qsd basically qemu-nbd in a new

Necessary when writing a client that wants to avoid unnecessary EINVAL errors from sending unaligned requests. At some point, we may want to add synchronous convenience API wrappers that do request splitting or read-modify-write to obey server constraints while still appearing to the library client as accepting any possible request. But such a wrapper should only be synchronous and not copied to

v1 was posted here: https://www.redhat.com/archives/libguestfs/2019-September/thread.html#00337 v2: - Drop the first patch. - Hopefully fix the multiple issues with fork-safety and general behaviour on error paths. Note this requires execvpe for which there seems to be no equivalent on FreeBSD, except some kind of tedious path parsing (but can we assign to environ?) Rich.

Similar to the recent --mask-handshake command line added to nbdkit to test client fallbacks to crippled servers, it can be worth testing server fallbacks to crippled clients. And just as we have exposed whether the client will request structured replies, we can also expose whether the client will understand various handshake flags from the NBD protocol. Of course, we default to supporting all