similar to: [PATCH libnbd discussion only 0/5] api: Implement concurrent writer.

v1: https://www.redhat.com/archives/libguestfs/2019-June/msg00014.html I pushed a few bits which are uncontroversial. The main changes since v1 are: An extra patch removes the want_to_send / check for nbd_aio_is_ready in examples/threaded-reads-and-writes.c. This logic was wrong since commit 6af72b87 as was pointed out by Eric in his review. Comments and structure of

--- .gitignore | 1 + examples/Makefile.am | 12 + examples/concurrent-writer.c | 450 +++++++++++++++++++++++++++ examples/threaded-reads-and-writes.c | 2 +- 4 files changed, 464 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 30438c1..e4dad91 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ Makefile.in

--- .gitignore | 1 + examples/Makefile.am | 12 + examples/concurrent-writer.c | 450 +++++++++++++++++++++++++++++++++++ 3 files changed, 463 insertions(+) diff --git a/.gitignore b/.gitignore index 30438c1..e4dad91 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ Makefile.in /docs/libnbd-api.3 /docs/libnbd-api.pod /examples/batched-read-write

This isn't quite finished because not all of the tests or examples have been updated, but it demonstrates an idea: Should we forget about the concept of having multiple connections managed under a single handle? In this patch there is a single ‘struct nbd_handle *’ which manages a single state machine and connection (and therefore no nbd_connection). To connect to a multi-conn server you must

I still need to wire in the use of *_notify functions into nbdkit to prove whether it makes the code any faster or easier to maintain, but at least the added example shows one good use case for the new API. Eric Blake (6): api: Add nbd_aio_in_flight generator: Allow DEAD state actions to run generator: Allow Int64 in callbacks states: Prepare for aio notify callback api: Add new

v1 was here: https://www.redhat.com/archives/libguestfs/2019-July/thread.html#00231 The changes address everything that Eric picked up in his review of the first two patches. I have also added two more patches (4 and 5) which respectively fix docs and change int status -> unsigned status, as discussed. Passes make, check, check-valgrind. Rich.

This implements most of what I wrote here: https://www.redhat.com/archives/libguestfs/2019-July/msg00213.html

RFC because this is an API break, but we haven't declared stable API yet. If we like it, I'm working on using libnbd to implement the nbdkit-nbd plugin; knowing whether it is API version 0.1 or 0.2 will be useful. I also dabbled with allowing optional parameters in python, although my OCaml is weak enough that there may be cleaner ways to approach that. Eric Blake (4): api: Add flags

As before, but this one has working Python bindings. OCaml still TBD. Rich.

Since v1: - rebase to applied patches - split out support for Int in callbacks - sort of test that callbacks work in OCaml (see comment in patch 5) - rename API to nbd_pread_structured - expose error as explicit parameter to callback Eric Blake (5): generator: Allow Int in callbacks states: Wire in a read callback states: Add nbd_pread_structured API states: Add tests for

** INCOMPLETE ** This is the generator change as discussed on the list already. The Python and OCaml bindings are not yet done. It passes all [C only] tests and valgrind. Note that nbd_add_close_callback is inconsistent with other closure types because it passes the user_data parameter after the function. (This is not caused by the current patch, it was already inconsistent). We decided that

I've mentioned this topic before (in fact, the idea of adding NBD_CMD_FLAG_DF was first mentioned at [1]), but finally finished enough of an implementation to feel confident in posting it. I'd still like to add something under examples/ that uses the new API to implement strict checking of a server's structured replies read implementation (ensure that a server never sends data after

These are some simplifications to the generator. They don't probably make much sense on their own, but they are preparatory to better handling of enums, and or'd lists of flags. Rich.

I think I've addressed everything that was raised in review. Some of the highlights: - Callbacks should be freed reliably along all exit paths. - There's a simple test of closure lifetimes. - I've tried to use VALID|FREE in all the places where I'm confident that it's safe and correct to do. There may be more places. Note this is an optimization and shouldn't

Posting now that I got something to compile (at the expense of breaking OCaml bindings), but I'm open to ideas on how to improve it. Eric Blake (2): generator: Tweak print_c_arg_list to take alternate first arg RFC: generator: Handle shared callbacks in Python generator/generator | 556 ++++++++++++++++++++++---------------------- 1 file changed, 280 insertions(+), 276 deletions(-) --

This works. I'm in the middle of testing whether there is any noticable benefit. Rich.

On 7/24/19 7:17 AM, Richard W.M. Jones wrote: > In preparation for closure lifetimes, split up the Closure so it no > longer describes a list of closures, but a single callback. > > This changes the API because functions which take 2 or more closures > now pass a separate user_data for each one. > --- > docs/libnbd.pod | 3 +- >

Nir, BTW what are you using for performance testing? As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v). Instead I've hacked up a copy of this program from libnbd: https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and-writes.c so

Still not working, but I took the latest patch and: - rebased it against libnbd 1.0 - fixed it so it handles new args and cbargs The generator now runs without warnings. This patch doesn't handle optargs at all. In C these are converted to non-optional parameter. Rust doesn't (AFAIK) have optional or labelled arguments unfortunately. Rich.

On 5/22/19 4:50 AM, Richard W.M. Jones wrote: > If not using multi-conn then obviously the synchronous connection > calls ‘nbd_connect_unix’, ‘nbd_connect_tcp’ and ‘nbd_connect_command’ > should only return when the (one) connection object is connected. > > In the multi-conn case it's not very clear what these synchronous > calls should do. Previously I had it so that they