Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 0/6] Introduce filters to nbdkit.
This patch isn't complete (patch 6/6 isn't finished) so it's just
for
discussion, although it does compile and run.
This introduces to nbdkit a concept of "filters" which can be placed
in front of plugins to modify their behaviour. Some examples where
you might use filters:
* Serve a subset of the data, such as (offset, range) or a
single partition from a disk image.
* Inject delays or errors for testing clients.
* Implement "copy-on-write" (a feature found in other NBD servers).
Filters are implemented by allowing them to intercept methods before a
plugin gets them. For example to implement a read delay the filter
would register for a .pread hook which is implemented like this:
static int
delay_pread (void *handle, void *buf, uint32_t count, uint64_t offset,
int (*next) (void *data,
void *buf, uint32_t count, uint64_t offset),
void *data)
{
nanosleep (...);
return next (data, buf, count, offset); // calls next filter or plugin
}
...
static struct nbdkit_filter filter = {
...
.pread = delay_pread,
...
};
For the filters I want to write this works fine, but with two caveats:
(1) If new datapath methods are introduced then filters won't get to
see them by default. For example, if we modify offsets when calling
.pread and .pwrite, and then later the .zero method is added to
plugins, then existing filters won't modify the parameters of .zero
correctly (resulting in wrong data being zeroed).
(2) You cannot do anything more complex in one of these functions than
calling the single underlying plugin method, possibly modifying the
arguments. For example it's hard to see how a "qcow2 decoder"
filter
could be written since it would need to have full access to the plugin
methods, not just to the single method.
Unfortunately solving (1) & (2) makes the whole thing a lot more
complicated.
Rich.
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 1/6] main: Refactor is_short_name test into a function.
No change, just refactoring.
---
src/main.c | 23 +++++++++++++++--------
1 file changed, 15 insertions(+), 8 deletions(-)
diff --git a/src/main.c b/src/main.c
index 46dd08c..4eca859 100644
--- a/src/main.c
+++ b/src/main.c
@@ -62,8 +62,9 @@
#define FIRST_SOCKET_ACTIVATION_FD 3 /* defined by systemd ABI */
+static int is_short_name (const char *);
static char *make_random_fifo (void);
-static void open_plugin_so (const char *filename, int is_short_name);
+static void open_plugin_so (const char *filename, int short_name);
static void start_serving (void);
static void set_up_signals (void);
static void run_command (void);
@@ -198,7 +199,7 @@ main (int argc, char *argv[])
int option_index;
int help = 0, version = 0, dump_plugin = 0;
int tls_set_on_cli = 0;
- int is_short_name;
+ int short_name;
const char *filename;
char *p;
@@ -461,13 +462,12 @@ main (int argc, char *argv[])
* help/version/plugin information.
*/
filename = argv[optind++];
- is_short_name - strchr (filename, '.') == NULL && strchr
(filename, '/') == NULL;
+ short_name = is_short_name (filename);
/* Is there an executable script located in the plugindir?
* If so we simply execute it with the current command line.
*/
- if (is_short_name) {
+ if (short_name) {
size_t i;
struct stat statbuf;
CLEANUP_FREE char *script;
@@ -493,7 +493,7 @@ main (int argc, char *argv[])
}
}
- open_plugin_so (filename, is_short_name);
+ open_plugin_so (filename, short_name);
if (help) {
usage ();
@@ -570,6 +570,13 @@ main (int argc, char *argv[])
exit (EXIT_SUCCESS);
}
+/* Is it a name relative to the plugindir? */
+static int
+is_short_name (const char *filename)
+{
+ return strchr (filename, '.') == NULL && strchr (filename,
'/') == NULL;
+}
+
/* Implementation of '-U -' */
static char *
make_random_fifo (void)
@@ -603,7 +610,7 @@ make_random_fifo (void)
}
static void
-open_plugin_so (const char *name, int is_short_name)
+open_plugin_so (const char *name, int short_name)
{
char *filename = (char *) name;
int free_filename = 0;
@@ -611,7 +618,7 @@ open_plugin_so (const char *name, int is_short_name)
struct nbdkit_plugin *(*plugin_init) (void);
char *error;
- if (is_short_name) {
+ if (short_name) {
/* Short names are rewritten relative to the plugindir. */
if (asprintf (&filename,
"%s/nbdkit-%s-plugin.so", plugindir, name) == -1) {
--
2.15.1
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 2/6] Introduce filters.
Filters can be placed in front of plugins to modify their behaviour.
This commit introduces the <nbdkit-filter.h> header file, the manual
page, the ‘filterdir’ directory (like ‘plugindir’), and the ‘filters/’
source directory which will contain both example and real filters.
---
Makefile.am | 2 +-
TODO | 17 +-
configure.ac | 3 +-
docs/Makefile.am | 9 +-
docs/nbdkit-filter.pod | 478 ++++++++++++++++++++++++++++++++++++++++++++++++
docs/nbdkit-plugin.pod | 3 +-
docs/nbdkit.pod | 3 +-
filters/Makefile.am | 33 ++++
include/Makefile.am | 4 +-
include/nbdkit-filter.h | 132 +++++++++++++
include/nbdkit-plugin.h | 2 +
src/Makefile.am | 5 +-
src/main.c | 1 +
src/nbdkit.pc.in | 1 +
14 files changed, 671 insertions(+), 22 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index f3c88b0..9c5b4c3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -49,7 +49,7 @@ SUBDIRS = \
src
if HAVE_PLUGINS
-SUBDIRS += plugins
+SUBDIRS += plugins filters
endif
SUBDIRS += tests
diff --git a/TODO b/TODO
index 0c027e2..0955db7 100644
--- a/TODO
+++ b/TODO
@@ -34,10 +34,8 @@ nbdkit there is no compelling reason unless the result is
better than
qemu-nbd. For the majority of users it would be better if they were
directed to qemu-nbd for these use cases.
-Filters
--------
-
-It should be possible to layer filters over plugins to do things like:
+Suggestions for filters
+-----------------------
* adding artificial delays (see wdelay/rdelay options in the file
plugin)
@@ -50,17 +48,6 @@ It should be possible to layer filters over plugins to do
things like:
* export a single partition (like qemu-nbd -P)
-A possible syntax would be:
-
- nbdkit --filter=delay [--filter=...] file file=foo wdelay=10
-
-The filter(s) intercept all plugin calls and can either return, return
-an error, or pass the call down to the next layer in the stack (and
-eventually to the plugin). By intercepting the .config call the
-filter can process its own parameters from the command line (wdelay=10
-in the example above), and by intercepting the .pread, .pwrite methods
-the filter could inject the delaying behaviour.
-
Composing nbdkit
----------------
diff --git a/configure.ac b/configure.ac
index a2950f6..7032614 100644
--- a/configure.ac
+++ b/configure.ac
@@ -181,7 +181,7 @@ AS_IF([test "x$POD2MAN" != "xno"],[
AM_CONDITIONAL([HAVE_POD2MAN], [test "x$POD2MAN" != "xno"])
AC_ARG_ENABLE([plugins],
- [AS_HELP_STRING([--disable-plugins], [disable all bundled plugins])])
+ [AS_HELP_STRING([--disable-plugins], [disable all bundled plugins and
filters])])
AM_CONDITIONAL([HAVE_PLUGINS], [test "x$enable_plugins" !=
"xno"])
dnl Check for Perl, for embedding in the perl plugin.
@@ -512,6 +512,7 @@ AC_CONFIG_FILES([Makefile
plugins/tar/Makefile
plugins/vddk/Makefile
plugins/xz/Makefile
+ filters/Makefile
src/Makefile
src/nbdkit.pc
tests/Makefile])
diff --git a/docs/Makefile.am b/docs/Makefile.am
index 323f48d..d2330fb 100644
--- a/docs/Makefile.am
+++ b/docs/Makefile.am
@@ -33,6 +33,7 @@
EXTRA_DIST = \
nbdkit.pod \
nbdkit-plugin.pod
+ nbdkit-filter.pod
CLEANFILES = *~
@@ -40,7 +41,8 @@ if HAVE_POD2MAN
man_MANS = \
nbdkit.1 \
- nbdkit-plugin.3
+ nbdkit-plugin.3 \
+ nbdkit-filter.3
CLEANFILES += $(man_MANS)
nbdkit.1: nbdkit.pod
@@ -53,4 +55,9 @@ nbdkit-plugin.3: nbdkit-plugin.pod
if grep 'POD ERROR' $@.t; then rm $@.t; exit 1; fi && \
mv $@.t $@
+nbdkit-filter.3: nbdkit-filter.pod
+ $(POD2MAN) $(POD2MAN_ARGS) --section=3 --name=nbdkit-filter $< $@.t
&& \
+ if grep 'POD ERROR' $@.t; then rm $@.t; exit 1; fi && \
+ mv $@.t $@
+
endif
diff --git a/docs/nbdkit-filter.pod b/docs/nbdkit-filter.pod
new file mode 100644
index 0000000..5fcd5ed
--- /dev/null
+++ b/docs/nbdkit-filter.pod
@@ -0,0 +1,478 @@
+=encoding utf8
+
+=head1 NAME
+
+nbdkit-filter - How to write nbdkit filters
+
+=head1 SYNOPSIS
+
+ #include <nbdkit-filter.h>
+
+ static int
+ myfilter_config (const char *key, const char *value,
+ int (*next) (...), void *data)
+ {
+ if (strcmp (key, "myparameter") == 0) {
+ // ...
+ return 0;
+ }
+ else {
+ // pass through to next filter or plugin
+ return next (data, key, value);
+ }
+ }
+
+ static struct nbdkit_filter filter = {
+ .name = "filter",
+ .config = myfilter_config,
+ /* etc */
+ };
+
+ NBDKIT_REGISTER_FILTER(filter)
+
+When this has been compiled to a shared library, do:
+
+ nbdkit [--args ...] --filter=./myfilter.so plugin [key=value ...]
+
+When debugging, use the I<-fv> options:
+
+ nbdkit -fv --filter=./myfilter.so plugin [key=value ...]
+
+=head1 DESCRIPTION
+
+One or more nbdkit filters can be placed in front of an nbdkit plugin
+to modify the behaviour of the plugin. This manual page describes how
+to create an nbdkit filter.
+
+Filters can be used for example to limit requests to an offset/limit,
+add copy-on-write support, or inject delays or errors (for testing).
+
+Different filters can be stacked:
+
+ NBD ┌─────────┐ ┌─────────┐ ┌────────┐
+ client ───▶│ filter1 │───▶│ filter2 │── ─ ─ ──▶│ plugin │
+ request └─────────┘ └─────────┘ └────────┘
+
+Each filter intercepts plugin functions (see L<nbdkit-plugin(3)>) and
+can modify parameters, call the following function in the chain
+before, in the middle or after, or even short-cut the chain. As an
+example, to process its own parameters the filter can intercept the
+C<.config> method:
+
+ static int
+ myfilter_config (const char *key, const char *value,
+ int (*next) (...), void *data)
+ {
+ if (strcmp (key, "myparameter") == 0) {
+ // ...
+ // here you would handle this key, value
+ // ...
+ return 0;
+ }
+ else {
+ // pass through to next filter or plugin
+ return next (data, key, value);
+ }
+ }
+
+ static struct nbdkit_filter filter = {
+ // ...
+ .config = myfilter_config,
+ // ...
+ };
+
+The call to C<next (data, ...)> calls the next filter or plugin in the
+chain. In the example above any instances of C<myparameter=...> on
+the command line would not be seen by the plugin.
+
+To see example filters, take a look at the source of nbdkit, in the
+C<filters> directory.
+
+Filters must be written in C and must be fully thread safe.
+
+=head1 C<nbdkit-filter.h>
+
+All filters should start by including this header file:
+
+ #include <nbdkit-filter.h>
+
+=head1 C<struct nbdkit_filter>
+
+All filters must define and register one C<struct nbdkit_filter>,
+which contains the name of the filter and pointers to plugin methods
+that the filter wants to intercept.
+
+ static struct nbdkit_filter filter = {
+ .name = "filter",
+ .longname = "My Filter",
+ .description = "This is my great filter for nbdkit",
+ .config = myfilter_config,
+ /* etc */
+ };
+
+ NBDKIT_REGISTER_FILTER(filter)
+
+The C<.name> field is the name of the filter. This is the only field
+which is required.
+
+=head1 CALLBACKS
+
+C<struct nbdkit_filter> has some static fields describing the filter
+and optional callback functions which can be used to intercept plugin
+methods.
+
+The functions have extra C<next> and C<data> parameters which are
used
+to call the next filter or plugin in the chain. Note that if your
+filter registers a callback but in that callback it doesn't call the
+C<next> function then the corresponding method in the plugin will
+never be called.
+
+You can modify parameters when you call the C<next> function. However
+be careful when modifying strings because for some methods
+(eg. C<.config>) the plugin may save the string pointer that you pass
+along. So you may have to ensure that the string is not freed for the
+lifetime of the server.
+
+=head2 C<.name>
+
+ const char *name;
+
+This field (a string) is required, and B<must> contain only ASCII
+alphanumeric characters and be unique amongst all filters.
+
+=head2 C<.version>
+
+ const char *version;
+
+Filters may optionally set a version string which is displayed in help
+and debugging output.
+
+=head2 C<.longname>
+
+ const char *longname;
+
+An optional free text name of the filter. This field is used in error
+messages.
+
+=head2 C<.description>
+
+ const char *description;
+
+An optional multi-line description of the filter.
+
+=head2 C<.load>
+
+ void load (void);
+
+This is called once just after the filter is loaded into memory. You
+can use this to perform any global initialization needed by the
+filter.
+
+=head2 C<.unload>
+
+ void unload (void);
+
+This may be called once just before the filter is unloaded from
+memory. Note that it's not guaranteed that C<.unload> will always be
+called (eg. the server might be killed or segfault), so you should try
+to make the filter as robust as possible by not requiring cleanup.
+See also L<nbdkit-plugin(3)/SHUTDOWN>.
+
+=head2 C<.config>
+
+ int config (const char *key, const char *value,
+ int (*next) (void *data, const char *key, const char *value),
+ void *data);
+
+This intercepts the plugin C<.config> method and can be used by the
+filter to parse its own command line parameters. You should try to
+make sure that command line parameter keys that the filter uses do not
+conflict with ones that could be used by a plugin.
+
+If there is an error, C<.config_complete> should call
C<nbdkit_error>
+with an error message and return C<-1>.
+
+=head2 C<.config_complete>
+
+ int config_complete (int (*next) (void *data), void *data);
+
+This intercepts the plugin C<.config_complete> method and can be used
+to ensure that all parameters needed by the filter were supplied on
+the command line.
+
+If there is an error, C<.config_complete> should call
C<nbdkit_error>
+with an error message and return C<-1>.
+
+=head2 C<.config_help>
+
+ const char *config_help;
+
+This optional multi-line help message should summarize any
+C<key=value> parameters that it takes. It does I<not> need to
repeat
+what already appears in C<.description>.
+
+If the filter doesn't take any config parameters you should probably
+omit this.
+
+=head2 C<.open>
+
+ void *open (int readonly,
+ int (*next) (void *data, int readonly), void *data);
+
+This intercepts the plugin C<.open> method and can be used to allocate
+per-connection data structures needed by the filter. The handle
+(which is not the same as the plugin handle) is passed back to other
+filter callbacks and could be freed in the C<.close> callback.
+
+Note that the handle is completely opaque to nbdkit, but it must not
+be NULL.
+
+Unlike other callbacks, you should I<not> call C<return next (...)>
+in this callback. Instead you should check for errors by doing:
+
+ if (next (data, readonly) == -1) {
+ // error from plugin .open method
+ // next() already called nbdkit_error so just return NULL
+ return NULL;
+ }
+ // no error
+ return my_handle;
+
+If there is an error, C<.open> should call C<nbdkit_error> with an
+error message and return C<NULL>.
+
+=head2 C<.close>
+
+ void close (void *handle,
+ void (*next) (void *data), void *data);
+
+This intercepts the plugin C<.close> method
+
+This is called when the client closes the connection. It should clean
+up any per-connection resources used by the filter.
+
+=head2 C<.get_size>
+
+ int64_t get_size (void *handle,
+ int64_t (*next) (void *data), void *data);
+
+This intercepts the plugin C<.get_size> method and can be used to read
+or modify the apparent size of the block device that the NBD client
+will see.
+
+The returned size must be E<ge> 0. If there is an error,
C<.get_size>
+should call C<nbdkit_error> with an error message and return C<-1>.
+
+=head2 C<.can_write>
+
+=head2 C<.can_flush>
+
+=head2 C<.is_rotational>
+
+=head2 C<.can_trim>
+
+ int can_write (void *handle,
+ int (*next) (void *data), void *data);
+
+ int can_flush (void *handle,
+ int (*next) (void *data), void *data);
+
+ int is_rotational (void *handle,
+ int (*next) (void *data), void *data);
+
+ int can_trim (void *handle,
+ int (*next) (void *data), void *data);
+
+These intercept the corresponding plugin methods.
+
+If there is an error, the callback should call C<nbdkit_error> with an
+error message and return C<-1>.
+
+=head2 C<.pread>
+
+ int pread (void *handle, void *buf, uint32_t count, uint64_t offset,
+ int (*next) (void *data,
+ void *buf, uint32_t count, uint64_t offset),
+ void *data);
+
+This intercepts the plugin C<.pread> method and can be used to read or
+modify data read by the plugin.
+
+If there is an error (including a short read which couldn't be
+recovered from), C<.pread> should call C<nbdkit_error> with an
error
+message B<and> set C<errno>, then return C<-1>.
+
+=head2 C<.pwrite>
+
+ int pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset,
+ int (*next) (void *data,
+ const void *buf, uint32_t count, uint64_t offset),
+ void *data);
+
+This intercepts the plugin C<.pwrite> method and can be used to modify
+data written by the plugin.
+
+If there is an error (including a short write which couldn't be
+recovered from), C<.pwrite> should call C<nbdkit_error> with an
error
+message B<and> set C<errno>, then return C<-1>.
+
+=head2 C<.flush>
+
+ int flush (void *handle,
+ int (*next) (void *data), void *data);
+
+This intercepts the plugin C<.flush> method and can be used to modify
+flush requests.
+
+If there is an error, C<.flush> should call C<nbdkit_error> with an
+error message B<and> set C<errno>, then return C<-1>.
+
+=head2 C<.trim>
+
+ int trim (void *handle, uint32_t count, uint64_t offset,
+ int (*next) (void *data, uint32_t count, uint64_t offset),
+ void *data);
+
+This intercepts the plugin C<.trim> method and can be used to modify
+trim requests.
+
+If there is an error, C<.trim> should call C<nbdkit_error> with an
+error message B<and> set C<errno>, then return C<-1>.
+
+=head2 C<.zero>
+
+ int zero (void *handle, uint32_t count, uint64_t offset, int may_trim,
+ int (*next) (void *data,
+ uint32_t count, uint64_t offset, int may_trim),
+ void *data);
+
+This intercepts the plugin C<.zero> method and can be used to modify
+zero requests.
+
+If there is an error, C<.zero> should call C<nbdkit_error> with an
+error message B<and> set C<errno>, then return C<-1>.
+
+=head1 THREADS
+
+Because filters can be mixed and used with any plugin and thus any
+threading model supported by L<nbdkit-plugin(3)>, filters must be
+thread safe. They must be able to handle concurrent requests even on
+the same handle.
+
+Filters may have to use pthread primitives like mutexes to achieve
+this.
+
+=head1 DEBUGGING
+
+Run the server with I<-f> and I<-v> options so it doesn't fork
and you
+can see debugging information:
+
+ nbdkit -fv --filter=./myfilter.so plugin [key=value [key=value [...]]]
+
+To print debugging information from within the filter, call
+C<nbdkit_debug>, which has the following prototype and works like
+L<printf(3)>:
+
+ void nbdkit_debug (const char *fs, ...);
+ void nbdkit_vdebug (const char *fs, va_list args);
+
+For convenience, C<nbdkit_debug> preserves the value of C<errno>.
+Note that C<nbdkit_debug> only prints things when the server is in
+verbose mode (I<-v> option).
+
+=head1 INSTALLING THE FILTER
+
+The filter is a C<*.so> file and possibly a manual page. You can of
+course install the filter C<*.so> file wherever you want, and users
+will be able to use it by running:
+
+ nbdkit --filter=/path/to/filter.so plugin [args]
+
+However B<if> the shared library has a name of the form
+C<nbdkit-I<name>-filter.so> B<and if> the library is
installed in the
+C<$filterdir> directory, then users can be run it by only typing:
+
+ nbdkit --filter=name plugin [args]
+
+The location of the C<$filterdir> directory is set when nbdkit is
+compiled and can be found by doing:
+
+ nbdkit --dump-config
+
+If using the pkg-config/pkgconf system then you can also find the
+filter directory at compile time by doing:
+
+ pkgconf nbdkit --variable=filterdir
+
+=head1 PKG-CONFIG/PKGCONF
+
+nbdkit provides a pkg-config/pkgconf file called C<nbdkit.pc> which
+should be installed on the correct path when the nbdkit development
+environment is installed. You can use this in autoconf
+F<configure.ac> scripts to test for the development environment:
+
+ PKG_CHECK_MODULES([NBDKIT], [nbdkit >= 1.2.3])
+
+The above will fail unless nbdkit E<ge> 1.2.3 and the header file is
+installed, and will set C<NBDKIT_CFLAGS> and C<NBDKIT_LIBS>
+appropriately for compiling filters.
+
+You can also run pkg-config/pkgconf directly, for example:
+
+ if ! pkgconf nbdkit --exists; then
+ echo "you must install the nbdkit development environment"
+ exit 1
+ fi
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-plugin(1)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2013-2018 Red Hat Inc.
+
+=head1 LICENSE
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+=over 4
+
+=item *
+
+Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+=item *
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+=item *
+
+Neither the name of Red Hat nor the names of its contributors may be
+used to endorse or promote products derived from this software without
+specific prior written permission.
+
+=back
+
+THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS''
AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
diff --git a/docs/nbdkit-plugin.pod b/docs/nbdkit-plugin.pod
index 9abf75f..3cafc42 100644
--- a/docs/nbdkit-plugin.pod
+++ b/docs/nbdkit-plugin.pod
@@ -692,6 +692,7 @@ and then users will be able to run it like this:
=head1 SEE ALSO
L<nbdkit(1)>,
+L<nbdkit-filter(3)>,
L<nbdkit-example1-plugin(1)>,
L<nbdkit-example2-plugin(1)>,
L<nbdkit-example3-plugin(1)>,
@@ -711,7 +712,7 @@ Pino Toscano
=head1 COPYRIGHT
-Copyright (C) 2013-2017 Red Hat Inc.
+Copyright (C) 2013-2018 Red Hat Inc.
=head1 LICENSE
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index 1687ac9..3b37db8 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -856,6 +856,7 @@ L</SOCKET ACTIVATION>.
Other nbdkit manual pages:
L<nbdkit-plugin(3)>,
+L<nbdkit-filter(3)>,
L<nbdkit-curl-plugin(1)>,
L<nbdkit-example1-plugin(1)>,
L<nbdkit-example2-plugin(1)>,
@@ -895,7 +896,7 @@ Pino Toscano
=head1 COPYRIGHT
-Copyright (C) 2013-2017 Red Hat Inc.
+Copyright (C) 2013-2018 Red Hat Inc.
=head1 LICENSE
diff --git a/filters/Makefile.am b/filters/Makefile.am
new file mode 100644
index 0000000..ed1580b
--- /dev/null
+++ b/filters/Makefile.am
@@ -0,0 +1,33 @@
+# nbdkit
+# Copyright (C) 2013-2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+#SUBDIRS diff --git a/include/Makefile.am b/include/Makefile.am
index 7d54215..deccc6b 100644
--- a/include/Makefile.am
+++ b/include/Makefile.am
@@ -30,4 +30,6 @@
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
-include_HEADERS = nbdkit-plugin.h
+include_HEADERS = \
+ nbdkit-plugin.h \
+ nbdkit-filter.h
diff --git a/include/nbdkit-filter.h b/include/nbdkit-filter.h
new file mode 100644
index 0000000..789bbb0
--- /dev/null
+++ b/include/nbdkit-filter.h
@@ -0,0 +1,132 @@
+/* nbdkit
+ * Copyright (C) 2013-2017 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* See nbdkit-filter(3) for documentation and how to write a filter. */
+
+#ifndef NBDKIT_FILTER_H
+#define NBDKIT_FILTER_H
+
+/* This header also defines some useful functions like nbdkit_debug
+ * and nbdkit_parse_size which are appropriate for filters to use.
+ */
+#include <nbdkit-plugin.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define NBDKIT_FILTER_API_VERSION 1
+
+struct nbdkit_filter {
+ /* Do not set these fields directly; use NBDKIT_REGISTER_FILTER.
+ * They exist so that we can support filters compiled against
+ * one version of the header with a runtime compiled against a
+ * different version with more (or fewer) fields.
+ */
+ uint64_t _struct_size;
+ int _api_version;
+
+ /* New fields will only be added at the end of the struct. */
+ const char *name;
+ const char *longname;
+ const char *version;
+ const char *description;
+
+ void (*load) (void);
+ void (*unload) (void);
+
+ int (*config) (const char *key, const char *value,
+ int (*next) (void *data, const char *key, const char *value),
+ void *data);
+ int (*config_complete) (int (*next) (void *data), void *data);
+ const char *config_help;
+
+ void * (*open) (int readonly,
+ int (*next) (void *data, int readonly), void *data);
+ void (*close) (void *handle,
+ void (*next) (void *data), void *data);
+
+ int64_t (*get_size) (void *handle,
+ int64_t (*next) (void *data), void *data);
+
+ int (*can_write) (void *handle,
+ int (*next) (void *data), void *data);
+ int (*can_flush) (void *handle,
+ int (*next) (void *data), void *data);
+ int (*is_rotational) (void *handle,
+ int (*next) (void *data), void *data);
+ int (*can_trim) (void *handle,
+ int (*next) (void *data), void *data);
+
+ int (*pread) (void *handle, void *buf, uint32_t count, uint64_t offset,
+ int (*next) (void *data,
+ void *buf, uint32_t count, uint64_t offset),
+ void *data);
+ int (*pwrite) (void *handle, const void *buf, uint32_t count, uint64_t
offset,
+ int (*next) (void *data,
+ const void *buf, uint32_t count, uint64_t
offset),
+ void *data);
+ int (*flush) (void *handle,
+ int (*next) (void *data), void *data);
+ int (*trim) (void *handle, uint32_t count, uint64_t offset,
+ int (*next) (void *data, uint32_t count, uint64_t offset),
+ void *data);
+ int (*zero) (void *handle, uint32_t count, uint64_t offset, int may_trim,
+ int (*next) (void *data,
+ uint32_t count, uint64_t offset, int may_trim),
+ void *data);
+};
+
+#ifndef NBDKIT_CXX_LANG_C
+#ifdef __cplusplus
+#define NBDKIT_CXX_LANG_C extern "C"
+#else
+#define NBDKIT_CXX_LANG_C /* nothing */
+#endif
+#endif
+
+#define NBDKIT_REGISTER_FILTER(filter) \
+ NBDKIT_CXX_LANG_C \
+ struct nbdkit_filter * \
+ filter_init (void) \
+ { \
+ (filter)._struct_size = sizeof (filter); \
+ (filter)._api_version = NBDKIT_API_VERSION; \
+ return &(filter); \
+ }
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* NBDKIT_FILTER_H */
diff --git a/include/nbdkit-plugin.h b/include/nbdkit-plugin.h
index 2ec3b15..13541e5 100644
--- a/include/nbdkit-plugin.h
+++ b/include/nbdkit-plugin.h
@@ -111,11 +111,13 @@ extern char *nbdkit_absolute_path (const char *path);
extern int64_t nbdkit_parse_size (const char *str);
extern int nbdkit_read_password (const char *value, char **password);
+#ifndef NBDKIT_CXX_LANG_C
#ifdef __cplusplus
#define NBDKIT_CXX_LANG_C extern "C"
#else
#define NBDKIT_CXX_LANG_C /* nothing */
#endif
+#endif
#define NBDKIT_REGISTER_PLUGIN(plugin) \
NBDKIT_CXX_LANG_C \
diff --git a/src/Makefile.am b/src/Makefile.am
index 12b9043..47d5f38 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -31,6 +31,7 @@
# SUCH DAMAGE.
plugindir = $(libdir)/nbdkit/plugins
+filterdir = $(libdir)/nbdkit/filters
sbin_PROGRAMS = nbdkit
@@ -46,13 +47,15 @@ nbdkit_SOURCES = \
sockets.c \
threadlocal.c \
utils.c \
- $(top_srcdir)/include/nbdkit-plugin.h
+ $(top_srcdir)/include/nbdkit-plugin.h \
+ $(top_srcdir)/include/nbdkit-filter.h
nbdkit_CPPFLAGS = \
-Dbindir=\"$(bindir)\" \
-Dlibdir=\"$(libdir)\" \
-Dmandir=\"$(mandir)\" \
-Dplugindir=\"$(plugindir)\" \
+ -Dfilterdir=\"$(filterdir)\" \
-Dsbindir=\"$(sbindir)\" \
-Dsysconfdir=\"$(sysconfdir)\" \
-I$(top_srcdir)/include
diff --git a/src/main.c b/src/main.c
index 4eca859..9b66d55 100644
--- a/src/main.c
+++ b/src/main.c
@@ -176,6 +176,7 @@ dump_config (void)
printf ("%s=%s\n", "mandir", mandir);
printf ("%s=%s\n", "name", PACKAGE_NAME);
printf ("%s=%s\n", "plugindir", plugindir);
+ printf ("%s=%s\n", "filterdir", filterdir);
printf ("%s=%s\n", "root_tls_certificates_dir",
root_tls_certificates_dir);
printf ("%s=%s\n", "sbindir", sbindir);
#ifdef HAVE_LIBSELINUX
diff --git a/src/nbdkit.pc.in b/src/nbdkit.pc.in
index cbb301d..fe8f511 100644
--- a/src/nbdkit.pc.in
+++ b/src/nbdkit.pc.in
@@ -3,6 +3,7 @@ exec_prefix=@exec_prefix@
libdir=@libdir@
includedir=@includedir@
plugindir=@libdir@/nbdkit/plugins
+filterdir=@libdir@/nbdkit/filters
Name: @PACKAGE_NAME@
Version: @PACKAGE_VERSION@
--
2.15.1
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 3/6] filters: Implement --filter parameter to load filter(s).
---
docs/nbdkit.pod | 21 ++++++++-
nbdkit.in | 17 ++++++-
src/internal.h | 4 +-
src/main.c | 53 +++++++++++++++++++++-
src/plugins.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
5 files changed, 223 insertions(+), 10 deletions(-)
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index 3b37db8..636eedc 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -7,7 +7,7 @@ nbdkit - A toolkit for creating NBD servers
=head1 SYNOPSIS
nbdkit [-e EXPORTNAME] [--exit-with-parent] [-f]
- [-g GROUP] [-i IPADDR]
+ [--filter=FILTER ...] [-g GROUP] [-i IPADDR]
[--newstyle] [--oldstyle] [-P PIDFILE] [-p PORT] [-r]
[--run CMD] [-s] [--selinux-label LABEL] [-t THREADS]
[--tls=off|on|require] [--tls-certificates /path/to/certificates]
@@ -119,6 +119,13 @@ not allowed with the oldstyle protocol.
I<Don't> fork into the background.
+=item B<--filter> FILTER
+
+Add a filter before the plugin. This option may be given one or more
+times to stack filters in front of the plugin. They are processed in
+the order they appear on the command line. See L</FILTERS> and
+L<nbdkit-filter(3)>.
+
=item B<-g> GROUP
=item B<--group> GROUP
@@ -354,6 +361,18 @@ languages. The file should be executable. For example:
(see L<nbdkit-perl-plugin(3)> for a full example).
+=head1 FILTERS
+
+One or more filters can be placed in front of an nbdkit plugin to
+modify the behaviour of the plugin, using the I<--filter> parameter.
+Filters can be used for example to limit requests to an offset/limit,
+add copy-on-write support, or inject delays or errors (for testing).
+
+Several existing filters are available in the C<$filterdir>. Use
+C<nbdkit --dump-config> to find the directory name.
+
+How to write filters is described in L<nbdkit-filter(3)>.
+
=head1 SOCKET ACTIVATION
nbdkit supports socket activation (sometimes called systemd socket
diff --git a/nbdkit.in b/nbdkit.in
index 20bc9c0..d4fe4e0 100644
--- a/nbdkit.in
+++ b/nbdkit.in
@@ -1,7 +1,7 @@
#!/bin/bash -
# @configure_input@
-# Copyright (C) 2017 Red Hat Inc.
+# Copyright (C) 2017-2018 Red Hat Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -79,6 +79,21 @@ while [ $# -gt 0 ]; do
shift
;;
+ # Filters can be rewritten if purely alphanumeric.
+ --filter)
+ args[$i]="--filter"
+ ((++i))
+ if [[ "$2" =~ ^[a-zA-Z0-9]+$ ]]; then
+ if [ -x "$b/filters/$2/.libs/nbdkit-$2-filter.so" ];
then
+
args[$i]="$b/filters/$2/.libs/nbdkit-$2-filter.so"
+ else
+ args[$i]="$2"
+ fi
+ fi
+ ((++i))
+ shift 2
+ ;;
+
# Anything else can be rewritten if it's purely alphanumeric,
# but there is only one module name so only rewrite once.
*)
diff --git a/src/internal.h b/src/internal.h
index 73bc09e..86cb0aa 100644
--- a/src/internal.h
+++ b/src/internal.h
@@ -1,5 +1,5 @@
/* nbdkit
- * Copyright (C) 2013-2017 Red Hat Inc.
+ * Copyright (C) 2013-2018 Red Hat Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -40,6 +40,7 @@
#include <pthread.h>
#include "nbdkit-plugin.h"
+#include "nbdkit-filter.h"
#ifdef __APPLE__
#define UNIX_PATH_MAX 104
@@ -142,6 +143,7 @@ extern int crypto_negotiate_tls (struct connection *conn,
int sockin, int sockou
#define debug nbdkit_debug
/* plugins.c */
+extern void filter_register (const char *_filename, void *_dl, struct
nbdkit_filter *(*filter_init) (void));
extern void plugin_register (const char *_filename, void *_dl, struct
nbdkit_plugin *(*plugin_init) (void));
extern void plugin_cleanup (void);
extern const char *plugin_name (void);
diff --git a/src/main.c b/src/main.c
index 9b66d55..f8c46b0 100644
--- a/src/main.c
+++ b/src/main.c
@@ -65,6 +65,7 @@
static int is_short_name (const char *);
static char *make_random_fifo (void);
static void open_plugin_so (const char *filename, int short_name);
+static void open_filter_so (const char *filename, int short_name);
static void start_serving (void);
static void set_up_signals (void);
static void run_command (void);
@@ -117,6 +118,7 @@ static const struct option long_options[] = {
{ "export", 1, NULL, 'e' },
{ "export-name",1, NULL, 'e' },
{ "exportname", 1, NULL, 'e' },
+ { "filter", 1, NULL, 0 },
{ "foreground", 0, NULL, 'f' },
{ "no-fork", 0, NULL, 'f' },
{ "group", 1, NULL, 'g' },
@@ -151,7 +153,7 @@ usage (void)
{
printf ("nbdkit [--dump-config] [--dump-plugin]\n"
" [-e EXPORTNAME] [--exit-with-parent] [-f]\n"
- " [-g GROUP] [-i IPADDR]\n"
+ " [--filter=FILTER ...] [-g GROUP] [-i IPADDR]\n"
" [--newstyle] [--oldstyle] [-P PIDFILE] [-p PORT]
[-r]\n"
" [--run CMD] [-s] [--selinux-label LABEL] [-t
THREADS]\n"
" [--tls=off|on|require] [--tls-certificates
/path/to/certificates]\n"
@@ -242,6 +244,9 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
#endif
}
+ else if (strcmp (long_options[option_index].name, "filter") ==
0) {
+ open_filter_so (optarg, is_short_name (optarg));
+ }
else if (strcmp (long_options[option_index].name, "run") == 0)
{
if (socket_activation) {
fprintf (stderr, "%s: cannot use socket activation with --run
flag\n",
@@ -571,7 +576,7 @@ main (int argc, char *argv[])
exit (EXIT_SUCCESS);
}
-/* Is it a name relative to the plugindir? */
+/* Is it a plugin or filter name relative to the plugindir/filterdir? */
static int
is_short_name (const char *filename)
{
@@ -654,6 +659,50 @@ open_plugin_so (const char *name, int short_name)
free (filename);
}
+static void
+open_filter_so (const char *name, int short_name)
+{
+ char *filename = (char *) name;
+ int free_filename = 0;
+ void *dl;
+ struct nbdkit_filter *(*filter_init) (void);
+ char *error;
+
+ if (short_name) {
+ /* Short names are rewritten relative to the filterdir. */
+ if (asprintf (&filename,
+ "%s/nbdkit-%s-filter.so", filterdir, name) == -1) {
+ perror ("asprintf");
+ exit (EXIT_FAILURE);
+ }
+ free_filename = 1;
+ }
+
+ dl = dlopen (filename, RTLD_NOW|RTLD_GLOBAL);
+ if (dl == NULL) {
+ fprintf (stderr, "%s: %s: %s\n", program_name, filename, dlerror
());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Initialize the filter. See dlopen(3) to understand C weirdness. */
+ dlerror ();
+ *(void **) (&filter_init) = dlsym (dl, "filter_init");
+ if ((error = dlerror ()) != NULL) {
+ fprintf (stderr, "%s: %s: %s\n", program_name, name, error);
+ exit (EXIT_FAILURE);
+ }
+ if (!filter_init) {
+ fprintf (stderr, "%s: %s: invalid filter_init\n", program_name,
name);
+ exit (EXIT_FAILURE);
+ }
+
+ /* Register the filter. */
+ filter_register (filename, dl, filter_init);
+
+ if (free_filename)
+ free (filename);
+}
+
static void
start_serving (void)
{
diff --git a/src/plugins.c b/src/plugins.c
index 9b5d2d5..3600293 100644
--- a/src/plugins.c
+++ b/src/plugins.c
@@ -1,5 +1,5 @@
/* nbdkit
- * Copyright (C) 2013 Red Hat Inc.
+ * Copyright (C) 2013-2018 Red Hat Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -44,8 +44,20 @@
#include <dlfcn.h>
#include "nbdkit-plugin.h"
+#include "nbdkit-filter.h"
#include "internal.h"
+/* If there are filters then ‘filters’ below will point to a linked
+ * list of filters in the order in which they must be applied to
+ * requests.
+ */
+struct filter {
+ struct filter *next;
+ char *filename;
+ void *dl;
+ struct nbdkit_filter filter;
+};
+
static pthread_mutex_t connection_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_mutex_t all_requests_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_rwlock_t unload_prevention_lock = PTHREAD_RWLOCK_INITIALIZER;
@@ -53,13 +65,111 @@ static pthread_rwlock_t unload_prevention_lock =
PTHREAD_RWLOCK_INITIALIZER;
/* Maximum read or write request that we will handle. */
#define MAX_REQUEST_SIZE (64 * 1024 * 1024)
-/* Currently the server can only load one plugin (see TODO). Hence we
- * can just use globals to store these.
+/* The server can only load one plugin. Hence we can just use globals
+ * to store these.
*/
+static struct filter *filters = NULL, *last_filter = NULL;
static char *filename;
static void *dl;
static struct nbdkit_plugin plugin;
+void
+filter_register (const char *_filename,
+ void *_dl, struct nbdkit_filter *(*filter_init) (void))
+{
+ const struct nbdkit_filter *_filter;
+ struct filter *f;
+ size_t i, len, size;
+
+ /* Allocate new entry in the linked list of filters. */
+ f = malloc (sizeof (*f));
+ if (f == NULL) {
+ perror ("malloc");
+ exit (EXIT_FAILURE);
+ }
+ if (last_filter)
+ last_filter->next = f;
+ else
+ filters = f;
+ last_filter = f;
+
+ f->next = NULL;
+ f->filename = strdup (_filename);
+ if (f->filename == NULL) {
+ perror ("strdup");
+ exit (EXIT_FAILURE);
+ }
+ f->dl = _dl;
+
+ debug ("registering filter %s", f->filename);
+
+ /* Call the initialization function which returns the address of the
+ * filter's own 'struct nbdkit_filter'.
+ */
+ _filter = filter_init ();
+ if (!_filter) {
+ fprintf (stderr, "%s: %s: filter registration function failed\n",
+ program_name, f->filename);
+ exit (EXIT_FAILURE);
+ }
+
+ /* Check for incompatible future versions. */
+ if (_filter->_api_version != 1) {
+ fprintf (stderr, "%s: %s: filter is incompatible with this version of
nbdkit (_api_version = %d)\n",
+ program_name, f->filename, _filter->_api_version);
+ exit (EXIT_FAILURE);
+ }
+
+ /* Since the filter might be much older than the current version of
+ * nbdkit, only copy up to the self-declared _struct_size of the
+ * filter and zero out the rest. If the filter is much newer then
+ * we'll only call the "old" fields.
+ */
+ size = sizeof (f->filter); /* our struct */
+ memset (&f->filter, 0, size);
+ if (size > _filter->_struct_size)
+ size = _filter->_struct_size;
+ memcpy (&f->filter, _filter, size);
+
+ /* Only filter.name is required. */
+ if (f->filter.name == NULL) {
+ fprintf (stderr, "%s: %s: filter must have a .name field\n",
+ program_name, f->filename);
+ exit (EXIT_FAILURE);
+ }
+
+ len = strlen (f->filter.name);
+ if (len == 0) {
+ fprintf (stderr, "%s: %s: filter.name field must not be empty\n",
+ program_name, f->filename);
+ exit (EXIT_FAILURE);
+ }
+ for (i = 0; i < len; ++i) {
+ if (!((f->filter.name[i] >= '0' &&
f->filter.name[i] <= '9') ||
+ (f->filter.name[i] >= 'a' &&
f->filter.name[i] <= 'z') ||
+ (f->filter.name[i] >= 'A' &&
f->filter.name[i] <= 'Z'))) {
+ fprintf (stderr, "%s: %s: filter.name ('%s') field must
contain only ASCII alphanumeric characters\n",
+ program_name, f->filename, f->filter.name);
+ exit (EXIT_FAILURE);
+ }
+ }
+
+ /* Copy the module's name into local storage, so that filter.name
+ * survives past unload.
+ */
+ if (!(f->filter.name = strdup (f->filter.name))) {
+ perror ("strdup");
+ exit (EXIT_FAILURE);
+ }
+
+ debug ("registered filter %s (name %s)", f->filename,
f->filter.name);
+
+ /* Call the on-load callback if it exists. */
+ debug ("%s: load", f->filename);
+ if (f->filter.load)
+ f->filter.load ();
+}
+
void
plugin_register (const char *_filename,
void *_dl, struct nbdkit_plugin *(*plugin_init) (void))
@@ -74,7 +184,7 @@ plugin_register (const char *_filename,
}
dl = _dl;
- debug ("registering %s", filename);
+ debug ("registering plugin %s", filename);
/* Call the initialization function which returns the address of the
* plugin's own 'struct nbdkit_plugin'.
@@ -150,7 +260,7 @@ plugin_register (const char *_filename,
exit (EXIT_FAILURE);
}
- debug ("registered %s (name %s)", filename, plugin.name);
+ debug ("registered plugin %s (name %s)", filename, plugin.name);
/* Call the on-load callback if it exists. */
debug ("%s: load", filename);
@@ -158,6 +268,22 @@ plugin_register (const char *_filename,
plugin.load ();
}
+static void
+cleanup_filters (struct filter *f)
+{
+ if (f) {
+ cleanup_filters (f->next);
+
+ debug ("%s: unload", f->filename);
+ if (f->filter.unload)
+ f->filter.unload ();
+
+ dlclose (f->dl);
+ free (f->filename);
+ free (f);
+ }
+}
+
void
plugin_cleanup (void)
{
@@ -176,6 +302,8 @@ plugin_cleanup (void)
free (filename);
filename = NULL;
+ cleanup_filters (filters);
+
pthread_rwlock_unlock (&unload_prevention_lock);
}
}
--
2.15.1
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 4/6] filters: Add nbdkit-offset-filter.
This very basic filter allows you to select an offset and range within
a plugin, for example:
nbdkit --filter=offset file file=foo offset=1M range=100M
which serves the byte range [ 1M .. 101M-1 ] from file ‘foo’.
---
TODO | 2 -
configure.ac | 1 +
filters/Makefile.am | 3 +-
filters/offset/Makefile.am | 62 +++++++++++++
filters/offset/nbdkit-offset-filter.pod | 99 ++++++++++++++++++++
filters/offset/offset.c | 154 ++++++++++++++++++++++++++++++++
6 files changed, 318 insertions(+), 3 deletions(-)
diff --git a/TODO b/TODO
index 0955db7..8eda0d7 100644
--- a/TODO
+++ b/TODO
@@ -44,8 +44,6 @@ Suggestions for filters
* copy-on-write, a popular feature in other servers
-* export a subset using offset/size
-
* export a single partition (like qemu-nbd -P)
Composing nbdkit
diff --git a/configure.ac b/configure.ac
index 7032614..4892dc4 100644
--- a/configure.ac
+++ b/configure.ac
@@ -513,6 +513,7 @@ AC_CONFIG_FILES([Makefile
plugins/vddk/Makefile
plugins/xz/Makefile
filters/Makefile
+ filters/offset/Makefile
src/Makefile
src/nbdkit.pc
tests/Makefile])
diff --git a/filters/Makefile.am b/filters/Makefile.am
index ed1580b..91fbe6c 100644
--- a/filters/Makefile.am
+++ b/filters/Makefile.am
@@ -30,4 +30,5 @@
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
-#SUBDIRS +SUBDIRS = \
+ offset
diff --git a/filters/offset/Makefile.am b/filters/offset/Makefile.am
new file mode 100644
index 0000000..f6e253c
--- /dev/null
+++ b/filters/offset/Makefile.am
@@ -0,0 +1,62 @@
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+EXTRA_DIST = nbdkit-offset-filter.pod
+
+CLEANFILES = *~
+
+filterdir = $(libdir)/nbdkit/filters
+
+filter_LTLIBRARIES = nbdkit-offset-filter.la
+
+nbdkit_offset_filter_la_SOURCES = \
+ offset.c \
+ $(top_srcdir)/include/nbdkit-filter.h
+
+nbdkit_offset_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include
+nbdkit_offset_filter_la_CFLAGS = \
+ $(WARNINGS_CFLAGS)
+nbdkit_offset_filter_la_LDFLAGS = \
+ -module -avoid-version -shared
+
+if HAVE_POD2MAN
+
+man_MANS = nbdkit-offset-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-offset-filter.1: nbdkit-offset-filter.pod
+ $(POD2MAN) $(POD2MAN_ARGS) --section=1 --name=`basename $@ .1` $< $@.t
&& \
+ if grep 'POD ERROR' $@.t; then rm $@.t; exit 1; fi && \
+ mv $@.t $@
+
+endif
diff --git a/filters/offset/nbdkit-offset-filter.pod
b/filters/offset/nbdkit-offset-filter.pod
new file mode 100644
index 0000000..c7f7bdf
--- /dev/null
+++ b/filters/offset/nbdkit-offset-filter.pod
@@ -0,0 +1,99 @@
+=encoding utf8
+
+=head1 NAME
+
+nbdkit-offset-filter - nbdkit offset filter
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=offset plugin offset=OFFSET range=LENGTH [plugin-args...]
+
+=head1 DESCRIPTION
+
+C<nbdkit-offset-filter> is a filter that limits requests to the byte
+range C<[offset .. range-1]> within another plugin.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<offset=OFFSET>
+
+The start offset.
+
+This parameter is required.
+
+=item B<range=LENGTH>
+
+The length of data to serve.
+
+This is optional. If not given then the range is served starting from
+the offset through to the end of the underlying file/device.
+
+=back
+
+Note it is an error if the range parameter is supplied and
+C<offset+range> is larger than the size of data served by the
+underlying plugin.
+
+=head1 EXAMPLE
+
+Using L<nbdkit-file-plugin(1)>, serve the C<100M> length range
+starting from C<1M> through to S<C<101M - 1 byte>> from the
file
+C<disk.img>:
+
+ nbdkit --filter=offset file file=disk.img offset=1M range=100M
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-file-plugin(1)>,
+L<nbdkit-filter(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2018 Red Hat Inc.
+
+=head1 LICENSE
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+=over 4
+
+=item *
+
+Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+=item *
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+=item *
+
+Neither the name of Red Hat nor the names of its contributors may be
+used to endorse or promote products derived from this software without
+specific prior written permission.
+
+=back
+
+THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS''
AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
diff --git a/filters/offset/offset.c b/filters/offset/offset.c
new file mode 100644
index 0000000..0b3b06f
--- /dev/null
+++ b/filters/offset/offset.c
@@ -0,0 +1,154 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+
+#include <nbdkit-filter.h>
+
+static int64_t offset = -1, range = -1;
+
+/* Called for each key=value passed on the command line. */
+static int
+offset_config (const char *key, const char *value,
+ int (*next) (void *data, const char *key, const char *value),
+ void *data)
+{
+ if (strcmp (key, "offset") == 0) {
+ offset = nbdkit_parse_size (value);
+ if (offset == -1)
+ return -1;
+ return 0;
+ }
+ else if (strcmp (key, "range") == 0) {
+ range = nbdkit_parse_size (value);
+ if (range == -1)
+ return -1;
+ return 0;
+ }
+ else
+ return next (data, key, value);
+}
+
+/* Check the user did pass both parameters. */
+static int
+offset_config_complete (int (*next) (void *data), void *data)
+{
+ if (offset == -1) {
+ nbdkit_error ("you must supply the offset parameter on the command
line");
+ return -1;
+ }
+
+ return next (data);
+}
+
+#define offset_config_help \
+ "offset=<OFFSET> (required) The start offset to serve.\n"
\
+ "range=<LENGTH> The total size to serve."
+
+/* Get the file size. */
+static int64_t
+offset_get_size (void *handle,
+ int64_t (*next) (void *data), void *data)
+{
+ int64_t real_size = next (data);
+
+ if (range >= 0) {
+ if (offset + range > real_size) {
+ nbdkit_error ("offset + range is larger than the real size of the
underlying file or device");
+ return -1;
+ }
+ return range;
+ }
+ else
+ return real_size - offset;
+}
+
+/* Read data. */
+static int
+offset_pread (void *handle, void *buf, uint32_t count, uint64_t offs,
+ int (*next) (void *data,
+ void *buf, uint32_t count, uint64_t offs),
+ void *data)
+{
+ return next (data, buf, count, offs + offset);
+}
+
+/* Write data. */
+static int
+offset_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offs,
+ int (*next) (void *data,
+ const void *buf, uint32_t count, uint64_t offs),
+ void *data)
+{
+ return next (data, buf, count, offs + offset);
+}
+
+/* Trim data. */
+static int
+offset_trim (void *handle, uint32_t count, uint64_t offs,
+ int (*next) (void *data, uint32_t count, uint64_t offs),
+ void *data)
+{
+ return next (data, count, offs + offset);
+}
+
+/* Zero data. */
+static int
+offset_zero (void *handle, uint32_t count, uint64_t offs, int may_trim,
+ int (*next) (void *data,
+ uint32_t count, uint64_t offs, int may_trim),
+ void *data)
+{
+ return next (data, count, offs + offset, may_trim);
+}
+
+static struct nbdkit_filter filter = {
+ .name = "offset",
+ .longname = "nbdkit offset filter",
+ .version = PACKAGE_VERSION,
+ .config = offset_config,
+ .config_complete = offset_config_complete,
+ .config_help = offset_config_help,
+ .get_size = offset_get_size,
+ .pread = offset_pread,
+ .pwrite = offset_pwrite,
+ .trim = offset_trim,
+ .zero = offset_zero,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
--
2.15.1
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 5/6] filters: Move rdelay/wdelay from file plugin to new delay filter.
Previously the file plugin supported ‘rdelay’ and ‘wdelay’ parameters
for injecting delays (for testing) into read and write requests. This
moves the functionality to a new delay filter so that it can be used
with any plugin.
---
TODO | 3 -
configure.ac | 1 +
filters/Makefile.am | 1 +
filters/delay/Makefile.am | 62 +++++++++++++
filters/delay/delay.c | 167 ++++++++++++++++++++++++++++++++++
filters/delay/nbdkit-delay-filter.pod | 88 ++++++++++++++++++
plugins/file/file.c | 76 +---------------
plugins/file/nbdkit-file-plugin.pod | 14 +--
8 files changed, 327 insertions(+), 85 deletions(-)
diff --git a/TODO b/TODO
index 8eda0d7..3ec45fd 100644
--- a/TODO
+++ b/TODO
@@ -37,9 +37,6 @@ directed to qemu-nbd for these use cases.
Suggestions for filters
-----------------------
-* adding artificial delays (see wdelay/rdelay options in the file
- plugin)
-
* injecting artificial errors for testing clients
* copy-on-write, a popular feature in other servers
diff --git a/configure.ac b/configure.ac
index 4892dc4..9376a2e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -513,6 +513,7 @@ AC_CONFIG_FILES([Makefile
plugins/vddk/Makefile
plugins/xz/Makefile
filters/Makefile
+ filters/delay/Makefile
filters/offset/Makefile
src/Makefile
src/nbdkit.pc
diff --git a/filters/Makefile.am b/filters/Makefile.am
index 91fbe6c..e4b2942 100644
--- a/filters/Makefile.am
+++ b/filters/Makefile.am
@@ -31,4 +31,5 @@
# SUCH DAMAGE.
SUBDIRS = \
+ delay
offset
diff --git a/filters/delay/Makefile.am b/filters/delay/Makefile.am
new file mode 100644
index 0000000..5b20b69
--- /dev/null
+++ b/filters/delay/Makefile.am
@@ -0,0 +1,62 @@
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+EXTRA_DIST = nbdkit-delay-filter.pod
+
+CLEANFILES = *~
+
+filterdir = $(libdir)/nbdkit/filters
+
+filter_LTLIBRARIES = nbdkit-delay-filter.la
+
+nbdkit_delay_filter_la_SOURCES = \
+ delay.c \
+ $(top_srcdir)/include/nbdkit-filter.h
+
+nbdkit_delay_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include
+nbdkit_delay_filter_la_CFLAGS = \
+ $(WARNINGS_CFLAGS)
+nbdkit_delay_filter_la_LDFLAGS = \
+ -module -avoid-version -shared
+
+if HAVE_POD2MAN
+
+man_MANS = nbdkit-delay-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-delay-filter.1: nbdkit-delay-filter.pod
+ $(POD2MAN) $(POD2MAN_ARGS) --section=1 --name=`basename $@ .1` $< $@.t
&& \
+ if grep 'POD ERROR' $@.t; then rm $@.t; exit 1; fi && \
+ mv $@.t $@
+
+endif
diff --git a/filters/delay/delay.c b/filters/delay/delay.c
new file mode 100644
index 0000000..24033cd
--- /dev/null
+++ b/filters/delay/delay.c
@@ -0,0 +1,167 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <time.h>
+
+#include <nbdkit-filter.h>
+
+static int rdelayms = 0; /* read delay (milliseconds) */
+static int wdelayms = 0; /* write delay (milliseconds) */
+
+static int
+parse_delay (const char *value)
+{
+ size_t len = strlen (value);
+ int r;
+
+ if (len > 2 && strcmp (&value[len-2], "ms") == 0) {
+ if (sscanf (value, "%d", &r) == 1)
+ return r;
+ else {
+ nbdkit_error ("cannot parse rdelay/wdelay milliseconds parameter:
%s",
+ value);
+ return -1;
+ }
+ }
+ else {
+ if (sscanf (value, "%d", &r) == 1)
+ return r * 1000;
+ else {
+ nbdkit_error ("cannot parse rdelay/wdelay seconds parameter:
%s",
+ value);
+ return -1;
+ }
+ }
+}
+
+static void
+delay (int ms)
+{
+ if (ms > 0) {
+ const struct timespec ts = {
+ .tv_sec = ms / 1000,
+ .tv_nsec = (ms * 1000000) % 1000000000
+ };
+ nanosleep (&ts, NULL);
+ }
+}
+
+static void
+read_delay (void)
+{
+ delay (rdelayms);
+}
+
+static void
+write_delay (void)
+{
+ delay (wdelayms);
+}
+
+/* Called for each key=value passed on the command line. */
+static int
+delay_config (const char *key, const char *value,
+ int (*next) (void *data, const char *key, const char *value),
+ void *data)
+{
+ if (strcmp (key, "rdelay") == 0) {
+ rdelayms = parse_delay (value);
+ if (rdelayms == -1)
+ return -1;
+ return 0;
+ }
+ else if (strcmp (key, "wdelay") == 0) {
+ wdelayms = parse_delay (value);
+ if (wdelayms == -1)
+ return -1;
+ return 0;
+ }
+ else
+ return next (data, key, value);
+}
+
+#define delay_config_help \
+ "rdelay=<NN>[ms] Read delay in
seconds/milliseconds.\n" \
+ "wdelay=<NN>[ms] Write delay in
seconds/milliseconds." \
+
+/* Read data. */
+static int
+delay_pread (void *handle, void *buf, uint32_t count, uint64_t offset,
+ int (*next) (void *data,
+ void *buf, uint32_t count, uint64_t offset),
+ void *data)
+{
+ read_delay ();
+ return next (data, buf, count, offset);
+}
+
+/* Write data. */
+static int
+delay_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset,
+ int (*next) (void *data,
+ const void *buf, uint32_t count, uint64_t offset),
+ void *data)
+{
+ write_delay ();
+ return next (data, buf, count, offset);
+}
+
+/* Zero data. */
+static int
+delay_zero (void *handle, uint32_t count, uint64_t offset, int may_trim,
+ int (*next) (void *data,
+ uint32_t count, uint64_t offset, int may_trim),
+ void *data)
+{
+ write_delay ();
+ return next (data, count, offset, may_trim);
+}
+
+static struct nbdkit_filter filter = {
+ .name = "delay",
+ .longname = "nbdkit delay filter",
+ .version = PACKAGE_VERSION,
+ .config = delay_config,
+ .config_help = delay_config_help,
+ .pread = delay_pread,
+ .pwrite = delay_pwrite,
+ .zero = delay_zero,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
diff --git a/filters/delay/nbdkit-delay-filter.pod
b/filters/delay/nbdkit-delay-filter.pod
new file mode 100644
index 0000000..10aba94
--- /dev/null
+++ b/filters/delay/nbdkit-delay-filter.pod
@@ -0,0 +1,88 @@
+=encoding utf8
+
+=head1 NAME
+
+nbdkit-delay-filter - nbdkit delay filter
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=delay plugin rdelay=SECS wdelay=SECS [plugin-args...]
+
+ nbdkit --filter=delay plugin rdelay=<NN>ms wdelay=<NN>ms
[plugin-args...]
+
+=head1 DESCRIPTION
+
+C<nbdkit-delay-filter> is a filter that delays read and write requests
+by some seconds or milliseconds. This is used to simulate a slow or
+remote server, or to test certain kinds of race conditions in Linux.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<rdelay=SECS>
+
+=item B<rdelay=E<lt>NNE<gt>ms>
+
+The optional read delay in seconds or milliseconds.
+
+=item B<wdelay=SECS>
+
+=item B<wdelay=E<lt>NNE<gt>ms>
+
+The optional write delay in seconds or milliseconds.
+
+=back
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-filter(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2018 Red Hat Inc.
+
+=head1 LICENSE
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+=over 4
+
+=item *
+
+Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+=item *
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+=item *
+
+Neither the name of Red Hat nor the names of its contributors may be
+used to endorse or promote products derived from this software without
+specific prior written permission.
+
+=back
+
+THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS''
AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
diff --git a/plugins/file/file.c b/plugins/file/file.c
index 4a91251..1fe4191 100644
--- a/plugins/file/file.c
+++ b/plugins/file/file.c
@@ -40,7 +40,6 @@
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
-#include <time.h>
#include <errno.h>
#include <nbdkit-plugin.h>
@@ -50,8 +49,6 @@
#endif
static char *filename = NULL;
-static int rdelayms = 0; /* read delay (milliseconds) */
-static int wdelayms = 0; /* write delay (milliseconds) */
static void
file_unload (void)
@@ -59,56 +56,6 @@ file_unload (void)
free (filename);
}
-static int
-parse_delay (const char *value)
-{
- size_t len = strlen (value);
- int r;
-
- if (len > 2 && strcmp (&value[len-2], "ms") == 0) {
- if (sscanf (value, "%d", &r) == 1)
- return r;
- else {
- nbdkit_error ("cannot parse rdelay/wdelay milliseconds parameter:
%s",
- value);
- return -1;
- }
- }
- else {
- if (sscanf (value, "%d", &r) == 1)
- return r * 1000;
- else {
- nbdkit_error ("cannot parse rdelay/wdelay seconds parameter:
%s",
- value);
- return -1;
- }
- }
-}
-
-static void
-delay (int ms)
-{
- if (ms > 0) {
- const struct timespec ts = {
- .tv_sec = ms / 1000,
- .tv_nsec = (ms * 1000000) % 1000000000
- };
- nanosleep (&ts, NULL);
- }
-}
-
-static void
-read_delay (void)
-{
- delay (rdelayms);
-}
-
-static void
-write_delay (void)
-{
- delay (wdelayms);
-}
-
/* Called for each key=value passed on the command line. This plugin
* only accepts file=<filename>, which is required.
*/
@@ -122,15 +69,10 @@ file_config (const char *key, const char *value)
if (!filename)
return -1;
}
- else if (strcmp (key, "rdelay") == 0) {
- rdelayms = parse_delay (value);
- if (rdelayms == -1)
- return -1;
- }
- else if (strcmp (key, "wdelay") == 0) {
- wdelayms = parse_delay (value);
- if (wdelayms == -1)
- return -1;
+ else if (strcmp (key, "rdelay") == 0 ||
+ strcmp (key, "wdelay") == 0) {
+ nbdkit_error ("add --filter=delay on the command line");
+ return -1;
}
else {
nbdkit_error ("unknown parameter '%s'", key);
@@ -157,9 +99,7 @@ file_config_complete (void)
}
#define file_config_help \
- "file=<FILENAME> (required) The filename to serve.\n" \
- "rdelay=<NN>[ms] Read delay in
seconds/milliseconds.\n" \
- "wdelay=<NN>[ms] Write delay in
seconds/milliseconds." \
+ "file=<FILENAME> (required) The filename to serve." \
/* The per-connection handle. */
struct handle {
@@ -241,8 +181,6 @@ file_pread (void *handle, void *buf, uint32_t count,
uint64_t offset)
{
struct handle *h = handle;
- read_delay ();
-
while (count > 0) {
ssize_t r = pread (h->fd, buf, count, offset);
if (r == -1) {
@@ -267,8 +205,6 @@ file_pwrite (void *handle, const void *buf, uint32_t count,
uint64_t offset)
{
struct handle *h = handle;
- write_delay ();
-
while (count > 0) {
ssize_t r = pwrite (h->fd, buf, count, offset);
if (r == -1) {
@@ -292,8 +228,6 @@ file_zero (void *handle, uint32_t count, uint64_t offset,
int may_trim)
#endif
int r = -1;
- write_delay ();
-
#ifdef FALLOC_FL_PUNCH_HOLE
if (may_trim) {
r = fallocate (h->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
diff --git a/plugins/file/nbdkit-file-plugin.pod
b/plugins/file/nbdkit-file-plugin.pod
index a2d1e6a..016513a 100644
--- a/plugins/file/nbdkit-file-plugin.pod
+++ b/plugins/file/nbdkit-file-plugin.pod
@@ -31,21 +31,13 @@ This parameter is required.
=item B<rdelay=E<lt>NNE<gt>ms>
-Delay reads for C<SECS> seconds or C<NN> milliseconds.
-This is used to simulate a slow or remote server, or to
-test certain kinds of race conditions in Linux.
-
-The default is no delay.
-
=item B<wdelay=SECS>
=item B<wdelay=E<lt>NNE<gt>ms>
-Delay writes for C<SECS> seconds or C<NN> milliseconds.
-This is used to simulate a slow or remote server, or to
-test certain kinds of race conditions in Linux.
-
-The default is no delay.
+These plugin parameters have been moved to the
+L<nbdkit-delay-filter(1)> filter. Modify the command line to add
+I<--filter=delay> in order to use these parameters.
=back
--
2.15.1
Richard W.M. Jones
2018-Jan-14 12:11 UTC
[Libguestfs] [PATCH nbdkit INCOMPLETE 6/6] filters: Implement chain of filters in front of ordinary plugin methods.
---
src/plugins.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 54 insertions(+), 7 deletions(-)
diff --git a/src/plugins.c b/src/plugins.c
index 3600293..085366b 100644
--- a/src/plugins.c
+++ b/src/plugins.c
@@ -406,8 +406,8 @@ plugin_dump_fields (void)
plugin.dump_plugin ();
}
-void
-plugin_config (const char *key, const char *value)
+static int
+final_plugin_config (const char *key, const char *value)
{
assert (dl);
@@ -422,21 +422,68 @@ plugin_config (const char *key, const char *value)
exit (EXIT_FAILURE);
}
- if (plugin.config (key, value) == -1)
- exit (EXIT_FAILURE);
+ return plugin.config (key, value);
+}
+
+static int
+filter_plugin_config (void *data, const char *key, const char *value)
+{
+ if (data == NULL)
+ return final_plugin_config (key, value);
+ else {
+ struct filter *f = data;
+
+ debug ("%s: config key=%s, value=%s", f->filename, key,
value);
+
+ if (f->filter.config)
+ if (f->filter.config (key, value, filter_plugin_config, f->next) ==
-1)
+ return -1;
+ return 0;
+ }
}
void
-plugin_config_complete (void)
+plugin_config (const char *key, const char *value)
+{
+ if (filter_plugin_config (filters, key, value) == -1)
+ exit (EXIT_FAILURE);
+}
+
+static int
+final_plugin_config_complete (void)
{
assert (dl);
debug ("%s: config_complete", filename);
if (!plugin.config_complete)
- return;
+ return 0;
+
+ return plugin.config_complete ();
+}
+
+static int
+filter_plugin_config_complete (void *data)
+{
+ if (data == NULL)
+ return final_plugin_config_complete ();
+ else {
+ struct filter *f = data;
+
+ debug ("%s: config_complete", f->filename);
- if (plugin.config_complete () == -1)
+ if (f->filter.config_complete)
+ if (f->filter.config_complete (filter_plugin_config_complete,
+ f->next) == -1)
+ return -1;
+ return 0;
+ }
+}
+
+void
+plugin_config_complete (void)
+{
+ if (filter_plugin_config_complete (filters) == -1)
exit (EXIT_FAILURE);
}
--
2.15.1
Richard W.M. Jones
2018-Jan-15 09:00 UTC
Re: [Libguestfs] [PATCH nbdkit INCOMPLETE 6/6] filters: Implement chain of filters in front of ordinary plugin methods.
Here's a fixed 6/6 patch. However I think this approach is wrong so I'm going to try to come up with an alternative way. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Eric Blake
2018-Jan-15 13:02 UTC
Re: [Libguestfs] [PATCH nbdkit INCOMPLETE 0/6] Introduce filters to nbdkit.
On 01/14/2018 06:11 AM, Richard W.M. Jones wrote:> This patch isn't complete (patch 6/6 isn't finished) so it's just for > discussion, although it does compile and run. > > This introduces to nbdkit a concept of "filters" which can be placed > in front of plugins to modify their behaviour. Some examples where > you might use filters: > > * Serve a subset of the data, such as (offset, range) or a > single partition from a disk image. > > * Inject delays or errors for testing clients. > > * Implement "copy-on-write" (a feature found in other NBD servers). > > Filters are implemented by allowing them to intercept methods before a > plugin gets them. For example to implement a read delay the filter > would register for a .pread hook which is implemented like this: > > static int > delay_pread (void *handle, void *buf, uint32_t count, uint64_t offset, > int (*next) (void *data, > void *buf, uint32_t count, uint64_t offset), > void *data)Interesting idea. I'm also wondering if we should start exposing flags from the client to the end user; in particular, NBD_CMD_FLAG_FUA is a useful optimization that avoids the overhead of a full-device flush if it is supported through the entire callstack - but that means we need both a new .pread callback variant that accepts a flag, as well as a filter signature that can pass the flag unchanged, as well as a way for a plugin to inform the main engine which flags it is willing to accept (at least with the NBD plugin, the ability to accept the FUA flag is dependent on the remote server, and it's nicer to have the main engine continue to emulate the needed flushes rather than having to duplicate the code in the NBD plugin for remote servers that don't accept the flag).> For the filters I want to write this works fine, but with two caveats: > > (1) If new datapath methods are introduced then filters won't get to > see them by default. For example, if we modify offsets when calling > .pread and .pwrite, and then later the .zero method is added to > plugins, then existing filters won't modify the parameters of .zero > correctly (resulting in wrong data being zeroed).Indeed, and my concern that we want new callbacks for handling flags in existing plugins is another instance of this.> > (2) You cannot do anything more complex in one of these functions than > calling the single underlying plugin method, possibly modifying the > arguments. For example it's hard to see how a "qcow2 decoder" filter > could be written since it would need to have full access to the plugin > methods, not just to the single method.Yeah, it's definitely conceivable that a filter will want to be able to access the underlying .read as part of implementing an intelligent .write.> > Unfortunately solving (1) & (2) makes the whole thing a lot more > complicated.In a lot of ways, it sounds like the development of filters in nbdkit will resemble the BDS filters in qemu by the time we get through with everything. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Eric Blake
2018-Jan-15 16:55 UTC
Re: [Libguestfs] [PATCH nbdkit INCOMPLETE 0/6] Introduce filters to nbdkit.
On 01/14/2018 06:11 AM, Richard W.M. Jones wrote:> This patch isn't complete (patch 6/6 isn't finished) so it's just for > discussion, although it does compile and run. > > This introduces to nbdkit a concept of "filters" which can be placed > in front of plugins to modify their behaviour. Some examples where > you might use filters: > > * Serve a subset of the data, such as (offset, range) or a > single partition from a disk image. > > * Inject delays or errors for testing clients. > > * Implement "copy-on-write" (a feature found in other NBD servers).How hard would it be to allow filters to control what features are advertised? For example, we've already proven that it is easy to expose features to the end client that are not present in the plugin (we advertise FUA support to end clients, because we can emulate it by calling the plugin's .flush at the right places; we also advertise write zero support because we can call the plugin's .pwrite as a fallback). But for testing purposes, it might be nice to permit the reverse: intentionally filter out a feature that the plugin supports, but which we do not want to advertise to the end client, for the purposes of testing the client's ability to fall back to other means in the absence of that feature. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Richard W.M. Jones
2018-Jan-15 17:16 UTC
Re: [Libguestfs] [PATCH nbdkit INCOMPLETE 0/6] Introduce filters to nbdkit.
On Mon, Jan 15, 2018 at 10:55:04AM -0600, Eric Blake wrote:> On 01/14/2018 06:11 AM, Richard W.M. Jones wrote: > > This patch isn't complete (patch 6/6 isn't finished) so it's just for > > discussion, although it does compile and run. > > > > This introduces to nbdkit a concept of "filters" which can be placed > > in front of plugins to modify their behaviour. Some examples where > > you might use filters: > > > > * Serve a subset of the data, such as (offset, range) or a > > single partition from a disk image. > > > > * Inject delays or errors for testing clients. > > > > * Implement "copy-on-write" (a feature found in other NBD servers). > > How hard would it be to allow filters to control what features are > advertised? For example, we've already proven that it is easy to expose > features to the end client that are not present in the plugin (we > advertise FUA support to end clients, because we can emulate it by > calling the plugin's .flush at the right places; we also advertise write > zero support because we can call the plugin's .pwrite as a fallback).I feel that the right way for us to support FUA is going to involve having a new .pwrite call with the additional flag, and I think we also will need a ‘.can_fua’ method so we don't need to overload can_flush as we do now. In that case any filter can simply overload those methods and modify it (in either direction, either adding or removing). Rich.> But for testing purposes, it might be nice to permit the reverse: > intentionally filter out a feature that the plugin supports, but which > we do not want to advertise to the end client, for the purposes of > testing the client's ability to fall back to other means in the absence > of that feature. > > -- > Eric Blake, Principal Software Engineer > Red Hat, Inc. +1-919-301-3266 > Virtualization: qemu.org | libvirt.org >-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org