search for: thread_model

...anged, 32 insertions(+), 68 deletions(-) diff --git a/docs/nbdkit-filter.pod b/docs/nbdkit-filter.pod index 4033789..86894f1 100644 --- a/docs/nbdkit-filter.pod +++ b/docs/nbdkit-filter.pod @@ -6,8 +6,6 @@ nbdkit-filter - how to write nbdkit filters #include <nbdkit-filter.h> - #define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL - static int myfilter_config (nbdkit_next_config *next, void *nxdata, const char *key, const char *value) @@ -107,24 +105,6 @@ should not expect to distribute filters separately from nbdkit. All filters should start by including this header fil...

Rich pointed out that making thread_model dynamic even for plugins makes some sense, so here is the code for doing it for 'sh'. I'm less confident on how to do it for OCaml and Rust (not to mention that those allow the plugin to already compile in their model, rather than the language binding glue forcing a model). The other l...

We do not promise API stability for non-C languages; this is an API break as follows: instead of calling 'NBDKit.register_plugin model plugin' with a static model, you can now add .thread_model :(unit -> thread_model) to plugin or default to PARALLEL. Since all existing OCaml plugins will have already thought about thread models, they can convert their existing model into the new plugin field (and thus, I don't feel too bad making PARALLEL the default, even if it is not always the...

Plugins cannot get rid of #define THREAD_MODEL (doing so would break API/ABI compatibility); however, we CAN add an optional callback for allowing a runtime reduction of the thread model. This can be especially useful for language bindings, as in the next patch, where the C glue code for the language has to cater to the maximum parallelism poss...

...CACHE work (since I touched the nocache filter); but can be rearranged if we think this series should go in first while that one undergoes any adjustments from review. Eric Blake (3): server: Allow filters to reduce thread model dynamically noparallel: Implement new filter filters: Use only .thread_model, not THREAD_MODEL docs/nbdkit-filter.pod | 45 ++++++---- filters/fua/nbdkit-fua-filter.pod | 1 + filters/nocache/nbdkit-nocache-filter.pod | 1 + filters/noextents/nbdkit-noextents-filter.pod | 1 + .../noparallel/nbdkit-noparallel-filter.pod | 66 +++...

...ocs/nbdkit-filter.pod +++ b/docs/nbdkit-filter.pod @@ -298,11 +298,18 @@ with an error message and return C<-1>. =head2 C<.get_ready> - int (*get_ready) (nbdkit_next_get_ready *next, void *nxdata); + int (*get_ready) (nbdkit_next_get_ready *next, void *nxdata, + int thread_model); This intercepts the plugin C<.get_ready> method and can be used by the filter to get ready to serve requests. +The C<thread_model> parameter informs the filter about the final +thread model chosen by nbdkit after considering the results of +C<.thread_model> of all filters in...

...s changed, 36 insertions(+), 7 deletions(-) diff --git a/docs/nbdkit-plugin.pod b/docs/nbdkit-plugin.pod index fe9ada87..9510253f 100644 --- a/docs/nbdkit-plugin.pod +++ b/docs/nbdkit-plugin.pod @@ -942,8 +942,16 @@ C<NBDKIT_REGISTER_PLUGIN>). Additionally, a plugin may implement the C<.thread_model> callback, called right after C<.config_complete> to make a runtime decision on which thread model to use. The nbdkit server chooses the most restrictive model between the plugin's -C<THREAD_MODEL>, the C<.thread_model> if present, and any restrictions -requested by filt...

...end, and then try to unlock the connection. Unfortunately the unlock operation has to check the thread model again which fails because the backend has gone away: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004070ce in unlock_connection () at locks.c:59 59 int thread_model = backend->thread_model (backend); [Current thread is 1 (Thread 0x7fab43243700 (LWP 6676))] (gdb) bt #0 0x00000000004070ce in unlock_connection () at locks.c:59 #1 0x00000000004061b1 in handle_single_connection (sockin=<optimized out>, sockout=<optimized out>) at conn...

The messages are not really useful to us, but they do bloat the debugging output of virt-v2v massively: nbdkit: python[1]: debug: acquire global request lock nbdkit: python[1]: debug: acquire per-connection request lock nbdkit: python[1]: debug: acquire unload prevention lock nbdkit: python[1]: debug: pwrite count=2097152 offset=4628414464 fua=0 nbdkit: python[1]: debug: release unload prevention

--- src/locks.c | 26 ++++++-------------------- 1 file changed, 6 insertions(+), 20 deletions(-) diff --git a/src/locks.c b/src/locks.c index bd8fd99..1724b5a 100644 --- a/src/locks.c +++ b/src/locks.c @@ -56,53 +56,39 @@ lock_init_thread_model (void) void lock_connection (void) { - if (thread_model <= NBDKIT_THREAD_MODEL_SERIALIZE_CONNECTIONS) { - debug ("acquire connection lock"); + if (thread_model <= NBDKIT_THREAD_MODEL_SERIALIZE_CONNECTIONS) pthread_mutex_lock (&connection_lock); - } } void unlo...

...;conn->status_lock); + if (conn->nworkers && + pthread_mutex_unlock (&conn->status_lock)) + abort (); return value; } diff --git a/server/locks.c b/server/locks.c index f4d6497..d70baf2 100644 --- a/server/locks.c +++ b/server/locks.c @@ -55,49 +55,59 @@ lock_init_thread_model (void) void lock_connection (void) { - if (thread_model <= NBDKIT_THREAD_MODEL_SERIALIZE_CONNECTIONS) - pthread_mutex_lock (&connection_lock); + if (thread_model <= NBDKIT_THREAD_MODEL_SERIALIZE_CONNECTIONS && + pthread_mutex_lock (&connection_lock)) + abort ()...

We do not promise API stability for non-C languages; this is an API break as follows: instead of calling plugin_init with a static model, you can now populate .thread_model in the Plugin struct, with a default to Parallel. As in C, the model is still chosen at .load time (at most, making it a function allows you to alter it based on configuration), and not something that can change per-connection. Since all existing Rust plugins will have already thought about threa...

On Fri, Aug 16, 2019 at 12:08:11PM -0500, Eric Blake wrote: > We do not promise API stability for non-C languages; this is an API > break as follows: instead of calling plugin_init with a static model, > you can now populate .thread_model in the Plugin struct, with a > default to Parallel. As in C, the model is still chosen at .load time > (at most, making it a function allows you to alter it based on > configuration), and not something that can change per-connection. > > Since all existing Rust plugins will have al...

On Mon, May 20, 2019 at 07:30:31AM -0500, Eric Blake wrote: > +=head2 C<.thread_model> > + > + int thread_model (void) > + > +This optional callback is called after all the configuration has been > +passed to the plugin. It can be used to force a stricter thread model > +based on configuration, compared to C<THREAD_MODEL>. See L</THREADS> > +bel...

...nbdkit OCaml interface - * Copyright (C) 2014-2019 Red Hat Inc. + * Copyright (C) 2014-2020 Red Hat Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are @@ -98,6 +98,8 @@ type 'a plugin = { thread_model : (unit -> thread_model) option; can_fast_zero : ('a -> bool) option; + + preconnect : (bool -> unit) option; } let default_callbacks = { @@ -145,6 +147,8 @@ let default_callbacks = { thread_model = None; can_fast_zero = None; + + preconnect = None; } external set_na...

When I added .can_FOO caching in 1.16, I missed the case that the sh plugin itself was calling .can_flush twice in some situations (in order to default .can_fua). Then right after, I regressed it to call .can_zero twice (in order to default .can_fast_zero). I also missed that .thread_model could use better caching, because at the time, I did not add testsuite coverage. Fix that now. Eric Blake (2): sh, eval: Cache .can_zero and .can_flush server: Better caching of .thread_model server/internal.h | 1 + server/connections.c | 8 ++- server/filters.c...

...dkit-filter.pod > @@ -298,11 +298,18 @@ with an error message and return C<-1>. > > =head2 C<.get_ready> > > - int (*get_ready) (nbdkit_next_get_ready *next, void *nxdata); > + int (*get_ready) (nbdkit_next_get_ready *next, void *nxdata, > + int thread_model); > > This intercepts the plugin C<.get_ready> method and can be used by the > filter to get ready to serve requests. > > +The C<thread_model> parameter informs the filter about the final > +thread model chosen by nbdkit after considering the results of > +C<...

On 5/20/19 8:36 AM, Richard W.M. Jones wrote: > On Mon, May 20, 2019 at 07:30:31AM -0500, Eric Blake wrote: >> +=head2 C<.thread_model> >> + >> + int thread_model (void) >> + >> +This optional callback is called after all the configuration has been >> +passed to the plugin. It can be used to force a stricter thread model >> +based on configuration, compared to C<THREAD_MODEL>. See L&l...

...of any existing Rust plugin), and done because I want to add fast_zero support to both languages as part of my upcoming fast zero series. Figuring out how to get extents working was hard enough that I punted that, still. Eric Blake (2): rust: Implement can_cache rust: Add support for dynamic .thread_model plugins/rust/nbdkit-rust-plugin.pod | 29 ++++++++++++++++++++++++----- plugins/rust/examples/ramdisk.rs | 8 ++++++-- plugins/rust/src/lib.rs | 23 ++++++++++++++++++++--- 3 files changed, 50 insertions(+), 10 deletions(-) -- 2.20.1

On Fri, Aug 02, 2019 at 02:26:11PM -0500, Eric Blake wrote: > Allow a plugin field to declare whether a parallel plugin can tolerate > windows where fds are not CLOEXEC, or must take precautions to avoid > leaking fds if the plugin may fork. For safety reasons, the flag > defaults to off, but many in-tree plugins can set it to on (most > commonly because they don't fork after