search for: get_ready

...ike this change. I think we were overloading the config_complete method before to do two different things (complete configuration; do any allocation/housekeeping necessary before we can start serving). The only questions in my mind are whether we want this before 1.18, and whether the name ("get_ready") is a good one. Rich.

Eric: Did you want to take this one any further? It might be one that we save for > 1.18: https://www.redhat.com/archives/libguestfs/2020-February/thread.html#00206 Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and also the last chance we get to set things up before the server forks. Of course such a change would not be material — .get_ready would always be run immediately after .config_complete — but it c...

On Sat, Feb 22, 2020 at 05:11:01AM -0600, Eric Blake wrote: > On 2/22/20 4:37 AM, Richard W.M. Jones wrote: > >Another thing I've been thinking about for some time is splitting > >.config_complete into .config_complete + .get_ready (new name TBD). > >At the moment .config_complete is both the place where we finish > >processing config, and also the last chance we get to set things up > >before the server forks. Of course such a change would not be > >material — .get_ready would always be run immediate...

The next patch wants to add a filter that will prevent DoS attacks from a plaintext client; to be successful, the filter must guarantee that nbdkit did not settle on SERIALIZE_CONNECTIONS. The easiest way to solve this is to expose the final thread model to .get_ready, which is after the point where .config_complete may have altered it, and before any connections are permitted. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/nbdkit-filter.pod | 9 ++++++++- include/nbdkit-filter.h | 3 ++- server/filters.c | 4 ++--...

..., Eric Blake wrote: > The next patch wants to add a filter that will prevent DoS attacks > from a plaintext client; to be successful, the filter must guarantee > that nbdkit did not settle on SERIALIZE_CONNECTIONS. The easiest way > to solve this is to expose the final thread model to .get_ready, which > is after the point where .config_complete may have altered it, and > before any connections are permitted. > > Signed-off-by: Eric Blake <eblake@redhat.com> > --- > docs/nbdkit-filter.pod | 9 ++++++++- > include/nbdkit-filter.h | 3 ++- > s...

If you have a plugin which either creates background threads itself or uses a library that creates background threads, it turns out you cannot create these in .get_ready (or earlier). The reason is that nbdkit forks when either daemonizing itself or using the --run option, and fork cancels all the background threads in the child process (the daemonized or captive nbdkit). The only good solution here is to add a new callback which I've called .after_fork which...

...te: >> The next patch wants to add a filter that will prevent DoS attacks >> from a plaintext client; to be successful, the filter must guarantee >> that nbdkit did not settle on SERIALIZE_CONNECTIONS. The easiest way >> to solve this is to expose the final thread model to .get_ready, which >> is after the point where .config_complete may have altered it, and >> before any connections are permitted. >> >> Signed-off-by: Eric Blake <eblake@redhat.com> >> --- >> docs/nbdkit-filter.pod | 9 ++++++++- >> include/nbdkit-fil...

...;The next patch wants to add a filter that will prevent DoS attacks > >>from a plaintext client; to be successful, the filter must guarantee > >>that nbdkit did not settle on SERIALIZE_CONNECTIONS. The easiest way > >>to solve this is to expose the final thread model to .get_ready, which > >>is after the point where .config_complete may have altered it, and > >>before any connections are permitted. > >> > >>Signed-off-by: Eric Blake <eblake@redhat.com> > >>--- > >> docs/nbdkit-filter.pod | 9 ++++++++- >...

VDDK plugin + --run was broken because of the following sequence of events: (1) After .config_complete, server redirects stdin/stdout to /dev/null. (2) Server then calls vddk_get_ready which reexecs. (3) We restart the server from the top, but with stdin/stdout redirected to /dev/null. So saved_stdin/saved_stdout save /dev/null. (4) run_command is called which "restores" /dev/null as stdin/stdout. (5) The output of the --run option is sent to /dev/null. In...

When compiling when an older nbdkit is installed, the build would fail because certain symbols such as .get_ready were not defined: ../../src/libguestfs.org/nbdkit/nbdkit.go:541:8: plugin.get_ready undefined (type _Ctype_struct_nbdkit_plugin has no field or method get_ready) This happens because we were using the installed <nbdkit-plugin.h> rather than the local copy. We don't want to modify the...

On Thu, Apr 23, 2020 at 05:58:25PM +0100, Daniel P. Berrangé wrote: > On Thu, Apr 23, 2020 at 05:55:14PM +0100, Richard W.M. Jones wrote: > > When compiling when an older nbdkit is installed, the build would fail > > because certain symbols such as .get_ready were not defined: > > > > ../../src/libguestfs.org/nbdkit/nbdkit.go:541:8: plugin.get_ready undefined (type _Ctype_struct_nbdkit_plugin has no field or method get_ready) > > > > This happens because we were using the installed <nbdkit-plugin.h> > > rather tha...

...riginal environ is sorted to make this more efficient, but the overhead of storing our replacement in a hash just to avoid the O(n^2) seems wasted. That argues that we should try to avoid invoking copy_environ in hot-spots (for example, in the sh plugin, can we get away with doing it once in .get_ready, rather than before every single callback?) hmm - looking at patch 8, you delayed things because of --run. We already know we have to call .config_complete before run_command(), but would it hurt if we reordered things to call run_command() prior to .get_ready?) > + len = strlen (key);...

...r *disk = NULL; > >+ const char *tmpdir; > >+ > >+ tmpdir = getenv ("TMPDIR"); > >+ if (!tmpdir) > >+ tmpdir = "/var/tmp"; > > Rather than calling getenv() for every client, should we > pre-populate a file-scope variable once during .get_ready? Is there a particular reason to do it in .get_ready instead of .load? In v3 I did it in .load Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual mac...

Patch 3 still needs tests added, but it is at least working from my simple command line tests. Eric Blake (3): server: Implement nbdkit_is_tls for use during .open server: Expose final thread_model to filter's .get_ready tlsdummy: New filter docs/nbdkit-filter.pod | 21 +- docs/nbdkit-plugin.pod | 34 ++- docs/nbdkit-tls.pod | 8 +- filters/log/nbdkit-log-filter.pod | 2 +- filters/tlsdummy/nbdkit-tlsdummy-filter.pod | 72 ++++++...

Version 2: https://www.redhat.com/archives/libguestfs/2020-April/thread.html#00166 Version 3 contains all changes discussed in the previous review. Rich.

...; is sorted to make this more efficient, but the overhead of storing > our replacement in a hash just to avoid the O(n^2) seems wasted. > That argues that we should try to avoid invoking copy_environ in > hot-spots (for example, in the sh plugin, can we get away with doing > it once in .get_ready, rather than before every single callback?) > > hmm - looking at patch 8, you delayed things because of --run. We > already know we have to call .config_complete before run_command(), > but would it hurt if we reordered things to call run_command() prior > to .get_ready?) Yes we c...

Version 1 was here: https://www.redhat.com/archives/libguestfs/2020-April/thread.html#00160 Version 2 side-steps the objections to the first patch by using a well-formed alternate nbdkit.pc file and running ordinary pkg-config against it, so any parsing of --cflags etc will be done by pkg-config. The first patch is essentially the same idea as:

On 6/22/20 10:49 AM, Richard W.M. Jones wrote: > If you have a plugin which either creates background threads itself or > uses a library that creates background threads, it turns out you > cannot create these in .get_ready (or earlier). The reason is that > nbdkit forks when either daemonizing itself or using the --run option, > and fork cancels all the background threads in the child process (the > daemonized or captive nbdkit). Or more precisely, the main thread after the fork cannot interact with the h...

...to file plugin + offset filter. From the user's point of view > the script would work a lot differently from other plugins. > > So no good ideas so far. How hard is it to write a tar filter instead of a tar plugin (similar to how we moved ext2 from plugin to filter)? - During .get_ready, we have to find some way to read the underlying plugin to feed a pipeline to tar to decode what the file contains (we can't just hand the file to tar, but instead have to feed it data through stdin; but the amount of work is no different: tar really does have to read the entire image durin...

On Thu, Apr 23, 2020 at 05:55:14PM +0100, Richard W.M. Jones wrote: > When compiling when an older nbdkit is installed, the build would fail > because certain symbols such as .get_ready were not defined: > > ../../src/libguestfs.org/nbdkit/nbdkit.go:541:8: plugin.get_ready undefined (type _Ctype_struct_nbdkit_plugin has no field or method get_ready) > > This happens because we were using the installed <nbdkit-plugin.h> > rather than the local copy. > &...