similar to: [libnbd PATCH v2 0/2] opt_list_meta_context

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit

This is a subset of my v2 posting, but limited to just the NBD_OPT_LIST handling. The biggest change since v2 is the addition of added unit testing in all four language bindings (C, python, ocaml, golang). The tests require nbdkit built from git on PATH, and may not be entirely idiomatic, but I at least validated that they catch issues (for example, adding an exit statement near the end of the

This adds coalescing of adjacent extents of the same type, as mentioned by Eric Blake in the commit message here: https://github.com/libguestfs/libnbd/commit/46072f6611f80245846a445766da071e457b00cd The patch series is rather long because it detours through adding the <vector.h> library from nbdkit into libnbd and replacing ad hoc uses of realloc, char ** etc in various places. Rich.

Right now, we require the user to supply potential metacontext names in advance, without knowing what the server actually supports until after the connection or nbd_opt_info call is complete. But the NBD protocol also supports a client being able to query what contexts a server supports, including where a client asks with a prefix and the server replies back with all answers that match the

As promised in my earlier thread on libnbd completion callback question. Eric Blake (2): generator: Refactor handling of closures in unlocked functions generator: Free closures on failure docs/libnbd.pod | 2 +- generator/C.ml | 48 +++++++++++------ generator/C.mli | 1 + lib/debug.c | 7 +-- lib/opt.c | 31 ++++++-----

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):

Use the just-added nbd_opt_list_meta_context() API to give more details about each export (matching what 'qemu-nbd --list' already does). Note that this requires some shuffling: listing meta exports requires being in opt mode, but is easiest to do in list_one_export(), which requires setting opt_mode in more code paths, and deferring ready mode until the last possible minute during

Now runs a complete set of tests, notably including the AIO test. File descriptors are passed in and out as plain ints (instead of *os.File) for a couple of reasons: (1) We have to pass the plain int to syscall.Select. (2) Turning an fd into an os.File causes golang to set the blocking flag which is deeply unhelpful. Rich.

Well, the golang changes (patch 1 and 2/3 of v1) were already committed, all that was left was the OCaml changes. I'm a lot happier with how things turned out with an UNKNOWN constructor in the OCaml variants. Eric Blake (3): tests: Enhance coverage of enum/flag range checking ocaml: Support unknown values for Enum/Flags ocaml: Typesafe returns for REnum/RFlags generator/OCaml.ml

Natural fallout after my recent testsuite additions that fixed a couple of ocaml bugs in the setters. However, on at least the OCaml code, I'm not sure what we should do if a newer libnbd ever returns a bit that an older NBD.mli was not expecting at the time the OCaml compiler ran (see below). I'm also not sure if there is a more efficient way to avoid outputting Val_FOO() converters for

v1 was here: https://listman.redhat.com/archives/libguestfs/2022-October/030216.html Since then, I've incorporated changes based on Rich's feedback: swap order of patches 2 and 3 less change in patch 1 (including no unsafe eval(%s) for --uri) in patch 2, include -c in list of snippets to store, and use dict of lambdas to map back to the desired action Eric Blake (3): nbdsh:

This continues the changes for adding NBD_OPT_LIST support. Now, instead of libnbd malloc'ing storage itself, the user passes a callback that can handle name/description pairs however it likes, and we get rid of the artificial cap at 10000 exports. However, the user will probably end up malloc'ing a list themselves, as we can't call nbd_set_export_name, or even request NBD_OPT_INFO

An incremental improvement on top of listing exports. I still think it's worth experimenting with revisiting how our API for list mode should actually work [1] (so that we can reuse a single connection for both grabbing the list and finally using NBD_OPT_GO), but this change was easier to whip together while still thinking about that. [1]

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2172516 v1: https://listman.redhat.com/archives/libguestfs/2023-April/031375.html In v2, move the declaration of the "p" helper variable next to the top of the function. Thanks! Laszlo Laszlo Ersek (2): generator/C: print_wrapper: use helper variable for permitted state check generator/C: lib/api.c: indent arg list 2

In v2: - now based on my proposal to add LIBNBD_SHUTDOWN_IMMEDIATE - four flags instead of two: STRICT_FLAGS is new (patch 4), and STRICT_BOUNDS is separate from STRICT_ZERO_SIZE (patch 5) - various refactorings for more shared code and less duplication Eric Blake (5): api: Add xxx_MASK constant for each Flags type generator: Refactor filtering of accepted OFlags api: Add

Hi, I scribbled a simple guestfs based program called guestfs-xfer with following synopsis: Usage: guest-xfer [options] [op] [diskimage] op = [ls|cat|write] options: -d, --guestdevice DEV guest device --blocksize BS blocksize [default: 1048576] -o, --offset OFF offset [default: 0] So eg. `cat /dev/urandom | guest-xfer -d /dev/sda

None are supported at present, so this always returns an empty list. --- docs/nbdkit-protocol.pod | 4 ++ server/protocol.h | 2 + server/connections.c | 84 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 90 insertions(+) diff --git a/docs/nbdkit-protocol.pod b/docs/nbdkit-protocol.pod index 68438fa..4c65e00 100644 --- a/docs/nbdkit-protocol.pod +++

This is the culmination of the previous patches' preparation work for using extended headers when possible. The new states in the state machine are copied extensively from our handling of OPT_STRUCTURED_REPLY. The next patch will then expose a new API nbd_opt_extended_headers() for manual control. At the same time I posted this patch, I had patches for qemu-nbd to support extended headers

--- plugins/python/nbdkit-python-plugin.pod | 19 ++++++- plugins/python/python.c | 75 +++++++++++++++++++++++++ tests/test-python-plugin.py | 8 ++- tests/test_python.py | 73 +++++++++++++++++++++++- 4 files changed, 169 insertions(+), 6 deletions(-) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod