similar to: [libnbd PATCH] API: Add nbd_set_opt_mode to expose NEGOTIATING state

It is finally time to introduce our first negotiating option command. With this change, we can now enter NEWSTYLE.START more than once; as such, it needs to know whether it is the first entry (proceed with gflags/cflags, TLS, and structured reply) or a later entry (all nbd_opt_* will cause an IssueCommand event to kick the state machine out of NEGOTIATING, at which point we want to jump to the

This continues the changes for adding NBD_OPT_LIST support. Now, instead of libnbd malloc'ing storage itself, the user passes a callback that can handle name/description pairs however it likes, and we get rid of the artificial cap at 10000 exports. However, the user will probably end up malloc'ing a list themselves, as we can't call nbd_set_export_name, or even request NBD_OPT_INFO

Right now, we require the user to supply potential metacontext names in advance, without knowing what the server actually supports until after the connection or nbd_opt_info call is complete. But the NBD protocol also supports a client being able to query what contexts a server supports, including where a client asks with a prefix and the server replies back with all answers that match the

On 8/11/20 12:12 PM, Richard W.M. Jones wrote: > I think this needs extra documentation in docs/libnbd.pod because it's > definitely not clear just from the rather thin manual page for > set_opt_mode how it works. docs/libnbd.pod is a good place for a > broader description of how it works. Yes, good idea. State-wise, the existing flow was: Created - Progress only by command

Squash this into opt_go? into opt_info? Standalone? Testing: why does python not throw an exception: $ ./run nbdsh Welcome to nbdsh, the shell for interacting with Network Block Device (NBD) servers. The ‘nbd’ module has already been imported and there is an open NBD handle called ‘h’. h.connect_tcp ("remote", "10809") # Connect to a remote server. h.get_size ()

I think this needs extra documentation in docs/libnbd.pod because it's definitely not clear just from the rather thin manual page for set_opt_mode how it works. docs/libnbd.pod is a good place for a broader description of how it works. IIUC let me try to explain: we get through as far as the end of group OPT_STRUCTURED_REPLY before we check the opt flag, and then the remainder of opt

This continues the changes for adding NBD_OPT_LIST support. Now, instead of libnbd malloc'ing storage itself, the user passes a callback that can handle name/description pairs however it likes, and we get rid of the artificial cap at 10000 exports. However, the user will probably end up malloc'ing a list themselves, as we can't call nbd_set_export_name, or even request NBD_OPT_INFO

On 7/20/20 9:43 AM, Richard W.M. Jones wrote: > A major missing feature of this library was the ability to list > exports from an NBD server. This implements the feature by adding a > new handle mode and additional functions for querying the list of > export names. > --- > .gitignore | 1 + > examples/Makefile.am | 14

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):

A major missing feature of this library was the ability to list exports from an NBD server. This implements the feature by adding a new handle mode and additional functions for querying the list of export names. --- .gitignore | 1 + examples/Makefile.am | 14 +++ examples/list-exports.c | 108 +++++++++++++++++++

Similar to the recent --mask-handshake command line added to nbdkit to test client fallbacks to crippled servers, it can be worth testing server fallbacks to crippled clients. And just as we have exposed whether the client will request structured replies, we can also expose whether the client will understand various handshake flags from the NBD protocol. Of course, we default to supporting all

Drop the SET designation, so that we can reuse these states for OPT_LIST_META_CONTEXT in the next patch. --- generator/Makefile.am | 4 +-- generator/state_machine.ml | 6 ++--- ...t.c => states-newstyle-opt-meta-context.c} | 26 +++++++++---------- .../states-newstyle-opt-structured-reply.c | 4 +-- generator/states-newstyle.c

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake

This is a subset of my v2 posting, but limited to just the NBD_OPT_LIST handling. The biggest change since v2 is the addition of added unit testing in all four language bindings (C, python, ocaml, golang). The tests require nbdkit built from git on PATH, and may not be entirely idiomatic, but I at least validated that they catch issues (for example, adding an exit statement near the end of the

This replaces 13/13 of my v2 series; and now that it has pretty good testsuite coverage and demonstrable performance improvement to nbdinfo, I'm going ahead and pushing this now. We may still want to add further nbd_opt_* commands for other fine-grained tuning of negotiation, but for now, I think things have stabilized on this end, and I can return to polishing .list_exports on the nbdkit

This is the culmination of the previous patches' preparation work for using extended headers when possible. The new states in the state machine are copied extensively from our handling of OPT_STRUCTURED_REPLY. The next patch will then expose a new API nbd_opt_extended_headers() for manual control. At the same time I posted this patch, I had patches for qemu-nbd to support extended headers

The NBD protocol is adding an extension for servers to report the initial state of the image (for now, whether it is sparse and/or reads as all zeroes). Time to expose this to clients, via the new API nbd_get_init_flags(). The patch requires refreshing the nbd-protocol.h file from a corresponding contemporary nbdkit commit. Testing is possible with recent enough qemu. --- .gitignore

When we moved to an enum instead of raw int for nbd_set_tls(), we should have also updated our code to prefer the enum values. While at it, improve the grammar of error messages (confusing since 632196ec, and copy-and-pasted into more locations since then). Fixes: 4488cf2a Thanks: Rich Jones --- Rich noticed this while reviewing the patch for today's CVE fix. It's not a show-stopper if

Proposal for new APIs to list exports. The general shape of the API can probably best be seen from the examples/list-exports.c example. Rich.

In v2: ack'ed preliminary patches have been pushed, and I've added a lot of testsuite coverage as well as putting the new API to use in nbdinfo. Eric Blake (2): api: Add nbd_opt_list_meta_context info: List available meta-contexts lib/internal.h | 1 + generator/API.ml | 84 ++++++++-