https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76
Already existing filters (readahead, cache) could be improved if
filters could open a background work thread or threads which connect
independently to the plugin. A proposed new filter (scan) cannot
really be written at all without this capability.
First of all the reason this can't be done today is because filters
are called with a next_ops structure which is only valid transiently
during the filter callback. It cannot safely be saved or passed to
another thread.
(https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#next-plugin)
The seemingly obvious implementation - which is what I tried today -
would be to let filters create background threads in .config_complete.
We would provide a filter API something like:
int nbdkit_open_connection (struct nbdkit_next_ops **next_ops,
void **nxdata,
/* needs a close function */);
It would return a next_ops and nxdata that the filter could then use
to make data calls from the background thread into the underlying
layers.
To ensure safe unloading of filters and plugins we would also need a
new filter callback (which I called .bg_kill) which must close all
background connections opened by the filter before returning.
I believe from my rough implementation that this is feasible. However
I also thought about another way we might do this: We might open a
loopback socket (eg. socketpair) which is passed to the filter and
which the filter connects to using libnbd. nbdkit internally would
treat this as if it was a regular external connection. Of course this
would require libnbd as a dependency, or disable filters / features if
not available.
Any thoughts on this?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
On 2/10/20 1:39 PM, Richard W.M. Jones wrote:> https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76 > > Already existing filters (readahead, cache) could be improved if > filters could open a background work thread or threads which connect > independently to the plugin. A proposed new filter (scan) cannot > really be written at all without this capability. > > First of all the reason this can't be done today is because filters > are called with a next_ops structure which is only valid transiently > during the filter callback. It cannot safely be saved or passed to > another thread. > (https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#next-plugin)I was independently thinking of fixing this: we want next_ops to be have a life equal to the connection itself, rather than stack-allocated. I'm hoping to post a patch for that shortly, as part of my experimentation with implementing ext2 as a filter instead of a plugin, where ext2 has the limitation that when writing a custom io_manager, you only get ONE spot where you can pass in an opaque pointer: our .prepare will have to pass a long-lived nxdata to ext2fs_open(). But that's only a partial solution towards the rest of your goal of having threading available.> > The seemingly obvious implementation - which is what I tried today - > would be to let filters create background threads in .config_complete. > We would provide a filter API something like: > > int nbdkit_open_connection (struct nbdkit_next_ops **next_ops, > void **nxdata, > /* needs a close function */); > > It would return a next_ops and nxdata that the filter could then use > to make data calls from the background thread into the underlying > layers. > > To ensure safe unloading of filters and plugins we would also need a > new filter callback (which I called .bg_kill) which must close all > background connections opened by the filter before returning.Seems reasonable.> > I believe from my rough implementation that this is feasible. However > I also thought about another way we might do this: We might open a > loopback socket (eg. socketpair) which is passed to the filter and > which the filter connects to using libnbd. nbdkit internally would > treat this as if it was a regular external connection. Of course this > would require libnbd as a dependency, or disable filters / features if > not available.There's also the question of how we would do compostion. For example, right now, the 'split' plugin basically acts as a compose operation over the 'file' plugin (you can concatenate more than one regular file), but more generically, we'd want some way to compose an arbitrary number of plugins and/or filters, where having a socketpair to each such composed child lets us handle more than just files. The 'nbd' plugin already has a dependency on libnbd, but allowing socketpair background threading would make this dependency extend to much more than that one plugin.> > Any thoughts on this? > > Rich. >-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Mon, Feb 10, 2020 at 01:52:25PM -0600, Eric Blake wrote:> There's also the question of how we would do compostion. For > example, right now, the 'split' plugin basically acts as a compose > operation over the 'file' plugin (you can concatenate more than one > regular file), but more generically, we'd want some way to compose > an arbitrary number of plugins and/or filters, where having a > socketpair to each such composed child lets us handle more than just > files.Since we allow plugins to have global variables, I suspect composition is impossible (unless there's some dlopen trick that I'm missing). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Mon, Feb 10, 2020 at 01:52:25PM -0600, Eric Blake wrote:> On 2/10/20 1:39 PM, Richard W.M. Jones wrote: > >https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76 > > > >Already existing filters (readahead, cache) could be improved if > >filters could open a background work thread or threads which connect > >independently to the plugin. A proposed new filter (scan) cannot > >really be written at all without this capability. > > > >First of all the reason this can't be done today is because filters > >are called with a next_ops structure which is only valid transiently > >during the filter callback. It cannot safely be saved or passed to > >another thread. > >(https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#next-plugin) > > I was independently thinking of fixing this: we want next_ops to be > have a life equal to the connection itself, rather than > stack-allocated. I'm hoping to post a patch for that shortly, as > part of my experimentation with implementing ext2 as a filter > instead of a plugin, where ext2 has the limitation that when writing > a custom io_manager, you only get ONE spot where you can pass in an > opaque pointer: our .prepare will have to pass a long-lived nxdata > to ext2fs_open().Yes I can see this could be tricky, especially in the presence of multiple threads serving a single connection. The only alternative way I can think to do it would be to use a pointer in thread-local storage which would would update with next_ops / nxdata each time. Do you have a preliminary patch for this yet, or do you want me to look at it today? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top