similar to: [libnbd PATCH] info: Add support for new 'qemu-nbd -A' qemu:allocation-depth

This is a rough-and-ready implementation of nbdinfo --map option, to display the allocation and other metadata of NBD exports. I only tested it lightly against nbdkit. It needs at least some tests. Command like these should work: nbdinfo --map nbd://localhost nbdinfo --map=qemu:dirty-bitmap nbd://localhost Rich.

This adds coalescing of adjacent extents of the same type, as mentioned by Eric Blake in the commit message here: https://github.com/libguestfs/libnbd/commit/46072f6611f80245846a445766da071e457b00cd The patch series is rather long because it detours through adding the <vector.h> library from nbdkit into libnbd and replacing ad hoc uses of realloc, char ** etc in various places. Rich.

Fixes bugs with JSON output in the previous patch, and adds some tests. Rich.

On 10/16/20 10:23 AM, Eric Blake wrote: > A rather trivial decoding; we may enhance it further if qemu extends > things to give an integer depth alongside its tri-state encoding. > --- > > I'll wait to push this to libnbd until the counterpart qemu patches > land upstream, although it looks like I've got positive review. Whoops, I accidentally pushed this before qemu

Otherwise, we get a failure due to Numerical result out of range. And for safety's sake, we are best capping our request to an aligned value, if the server insists on minimum alignment. Fixes: f3fd935c --- info/nbdinfo.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/info/nbdinfo.c b/info/nbdinfo.c index 1afdf98..2b22f51 100644 --- a/info/nbdinfo.c +++

On 9/26/20 11:36 AM, Richard W.M. Jones wrote: > --- > info/Makefile.am | 4 + > info/info-map-base-allocation-json.sh | 52 ++++++++++ > info/info-map-base-allocation.sh | 49 ++++++++++ > info/nbdinfo.c | 134 ++++++++++++++++++++++++-- > info/nbdinfo.pod | 39 +++++++- > 5 files changed, 271

--- info/Makefile.am | 4 + info/info-map-base-allocation-json.sh | 52 ++++++++++ info/info-map-base-allocation.sh | 49 ++++++++++ info/nbdinfo.c | 134 ++++++++++++++++++++++++-- info/nbdinfo.pod | 39 +++++++- 5 files changed, 271 insertions(+), 7 deletions(-) diff --git a/info/Makefile.am b/info/Makefile.am index

If the server fails, nbdinfo can write partial output before the error message (albeit on different channels). Here is an example: $ nbdkit eval open='echo EIO fail >&2; exit 1' --run 'nbdinfo --json "$uri"' { "protocol": "newstyle-fixed", "TLS": false, nbdkit: eval[1]: error: /tmp/nbdkitii3pZW/open: fail nbdkit: eval[1]:

An incremental improvement on top of listing exports. I still think it's worth experimenting with revisiting how our API for list mode should actually work [1] (so that we can reuse a single connection for both grabbing the list and finally using NBD_OPT_GO), but this change was easier to whip together while still thinking about that. [1]

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

When “nbdkit ... --run 'nbdinfo ...' > out” is expected to fail, we cannot be sure of the content of the "out" file. In particular nbdinfo produces its output incrementally, so in JSON mode there's usually a stray "{" in the output, and in non-JSON mode it usually prints the "protocol: " line. Thus it's not correct to test for the output file

On 9/29/20 6:23 AM, Richard W.M. Jones wrote: > This is not related to the current patch, but generally for libnbd: > > (1) Should we be testing interop separately for qemu-storage-daemon > (qsd), or is qsd basically qemu-nbd in a new wrapper so it's not worth > doing it? > > (2) I found a bug in the new nbdinfo behaviour: > > $ nbdkit -fv file dir=/scratch >

Place the tracing of return values after any generated 'out:' label, as it is important to see even the errors caused by calling a function in the wrong handle state. Fixes: 9df088bfa9 --- Multiple bugs found today, but only time to submit one patch so far. I found them all while trying to implement .list_exports in nbdkit. Bug 1: 'nbdinfo --list --json $uri' produces invalid

On 9/29/20 9:37 AM, Richard W.M. Jones wrote: > On Tue, Sep 29, 2020 at 06:51:06AM -0500, Eric Blake wrote: >> On 9/29/20 6:23 AM, Richard W.M. Jones wrote: >>> This is not related to the current patch, but generally for libnbd: >>> >>> (1) Should we be testing interop separately for qemu-storage-daemon >>> (qsd), or is qsd basically qemu-nbd in a new

On 10/7/23 11:27, Richard W.M. Jones wrote: > On Fri, Oct 06, 2023 at 10:18:09AM -0500, Eric Blake wrote: >> Adding a human-readable size for block constraints is useful. For: >> >> $ ./run nbdinfo -- [ nbdkit memory \ >> --filter=blocksize-policy blocksize-preferred=32k 1M ] | grep pref >> >> this changes pre-patch: >> block_size_preferred: 32768

Necessary when writing a client that wants to avoid unnecessary EINVAL errors from sending unaligned requests. At some point, we may want to add synchronous convenience API wrappers that do request splitting or read-modify-write to obey server constraints while still appearing to the library client as accepting any possible request. But such a wrapper should only be synchronous and not copied to

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit

In v2: ack'ed preliminary patches have been pushed, and I've added a lot of testsuite coverage as well as putting the new API to use in nbdinfo. Eric Blake (2): api: Add nbd_opt_list_meta_context info: List available meta-contexts lib/internal.h | 1 + generator/API.ml | 84 ++++++++-

This rounds up the remaining bugs that I originally identified in: https://www.redhat.com/archives/libguestfs/2020-July/msg00153.html Eric Blake (4): api: Permit export list APIs when Connected info: Support --list with serializing servers info: Fix --json output when list size != 1 info: Permit --size --json generator/API.ml | 6 +++--- info/info-list-json.sh | 9 +++++++++

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):