search for: thread_model_parallel

...n the plugin record passed to +C<NBDKit.register_plugin> is C<thread model>, which must return one of +the values in the table below. For more information on thread models, +see L<nbdkit-plugin(3)/THREADS>. If this optional function is not +provided, the thread model defaults to THREAD_MODEL_PARALLEL. Note that because of the garbage collector lock in OCaml, callbacks are never truly concurrent. diff --git a/plugins/ocaml/ocaml.c b/plugins/ocaml/ocaml.c index f664a7fb..01f4448f 100644 --- a/plugins/ocaml/ocaml.c +++ b/plugins/ocaml/ocaml.c @@ -72,6 +72,7 @@ static void remove_roots (void);...

...LIZE_ALL_REQUESTS", > + NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS); > + PyModule_AddIntConstant (m, "THREAD_MODEL_SERIALIZE_REQUESTS", > + NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS); > + PyModule_AddIntConstant (m, "THREAD_MODEL_PARALLEL", > + NBDKIT_THREAD_MODEL_PARALLEL); > + > + PyModule_AddIntConstant (m, "FLAG_MAY_TRIM", NBDKIT_FLAG_MAY_TRIM); > + PyModule_AddIntConstant (m, "FLAG_FUA", NBDKIT_FLAG_FUA); > + PyModule_AddIntConstant (m, "FLAG_REQ_O...

...+ NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS); > > > + PyModule_AddIntConstant (m, "THREAD_MODEL_SERIALIZE_REQUESTS", > > > + NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS); > > > + PyModule_AddIntConstant (m, "THREAD_MODEL_PARALLEL", > > > + NBDKIT_THREAD_MODEL_PARALLEL); > > > + > > > + PyModule_AddIntConstant (m, "FLAG_MAY_TRIM", NBDKIT_FLAG_MAY_TRIM); > > > + PyModule_AddIntConstant (m, "FLAG_FUA", NBDKIT_FLAG_FUA); > > &g...

...t;THREAD_MODEL_SERIALIZE_ALL_REQUESTS", + NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS); + PyModule_AddIntConstant (m, "THREAD_MODEL_SERIALIZE_REQUESTS", + NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS); + PyModule_AddIntConstant (m, "THREAD_MODEL_PARALLEL", + NBDKIT_THREAD_MODEL_PARALLEL); + + PyModule_AddIntConstant (m, "FLAG_MAY_TRIM", NBDKIT_FLAG_MAY_TRIM); + PyModule_AddIntConstant (m, "FLAG_FUA", NBDKIT_FLAG_FUA); + PyModule_AddIntConstant (m, "FLAG_REQ_ONE", NBDKIT_FLAG_R...

...ot;, > > + NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS); > > + PyModule_AddIntConstant (m, "THREAD_MODEL_SERIALIZE_REQUESTS", > > + NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS); > > + PyModule_AddIntConstant (m, "THREAD_MODEL_PARALLEL", > > + NBDKIT_THREAD_MODEL_PARALLEL); > > + > > + PyModule_AddIntConstant (m, "FLAG_MAY_TRIM", NBDKIT_FLAG_MAY_TRIM); > > + PyModule_AddIntConstant (m, "FLAG_FUA", NBDKIT_FLAG_FUA); > > + PyModule_AddIntCons...

Patch 2 certainly allows you to set the thread model. However patch 3 shows that if you set it to nbdkit.THREAD_MODEL_PARALLEL it will crash. If you look closely at the stack trace (attached below) you can see that ignoring threads which are in parts of nbdkit unrelated to Python: Thread 4: In pread, waiting in time.sleep(). This thread has released the GIL. Thread 2: Started to process a pread call but didn&...

...TWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +# Python plugin which uses the parallel thread model. Note that the +# cpython implementation of time.sleep releases the GIL (see +# Modules/timemodule.c:pysleep) + +import nbdkit +import time + +def thread_model(): + return nbdkit.THREAD_MODEL_PARALLEL + +def open(readonly): + return {} + +def get_size(h): + return 512 + +def pread(h, count, offset): + time.sleep(10) + return bytearray(count) -- 2.27.0

...RuntimeError("'transfer_url' parameter is required") > + > + > +def thread_model(): > + """ > + Using parallel model to speed up transfer with multiple connections to > + imageio server. > + """ > + return nbdkit.THREAD_MODEL_PARALLEL > + > + > +def open(readonly): > + """ > + Called once when plugin is loaded. We created a pool of connected clients > + that will be used for requests later. > + """ > + pool = queue.Queue() > + for i in range(params["...

...sfer_url"] is None: + raise RuntimeError("'transfer_url' parameter is required") + + +def thread_model(): + """ + Using parallel model to speed up transfer with multiple connections to + imageio server. + """ + return nbdkit.THREAD_MODEL_PARALLEL + + +def open(readonly): + """ + Called once when plugin is loaded. We created a pool of connected clients + that will be used for requests later. + """ + pool = queue.Queue() + for i in range(params["connections"]): + client = Imagei...

This is working for me now, although possibly only on Python 3.9. Dan suggested PyEval_InitThreads but that was deprecated in Python 3.7. Rich.

And fill out most of the missing bits of the API. Rich.

v1: https://www.redhat.com/archives/libguestfs/2019-November/msg00153.html v2: - Fix implementation of can_cache. - Add implementation of can_fua. - Add a very thorough test suite which tests every command + flag combination.

I'm still working on the interleaving (and Rich reminded me on IRC that we still don't have THREAD_MODEL_PARALLEL working anywhere yet, anyways). Since nbdkit doesn't really have a parallel plugin yet, my testing on that front will have to use qemu-nbd as the original server, as well as qemu-io as the driver (qemu-io's aio_read and aio_write commands can be used to trigger interleaved requests, and we...

v3 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00209.html In v4: - Rebase on top of current master. Includes various fixes and updates required because of Nir's patches that went into master. - Fix api_version() -> API_VERSION in patch 2 noted previously on the mailing list. Rich.

v2 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00163.html I pushed patch 1 (with spelling fix), patch 4 and patch 5 since those were previously ACKed on the list. Differences in v3: - Add error checking to PyModule_AddIntConstant. - Use API_VERSION constant instead of function. - Add max API version supported to --dump-plugin output. - Print API_VERSION selected by