Richard W.M. Jones
2018-Jul-31 19:55 UTC
[Libguestfs] [PATCH nbdkit 0/4] Add truncate and map filters.
This patch series proposes two new filters. * truncate: This can truncate, extend, round up or round down the size of a plugin/device. A typical usage is to fix the qemu problem that it can only handle devices which are a multiple of 512-bytes: nbdkit --filter=truncate random size=500 round-up=512 This will serve a virtual device with size 512 bytes. Reading from the last 12 bytes will return zeroes. And writing is permitted, provided you only write zeroes. An alternative might have been to extend the offset filter to deal with this, but that could have got quite clumsy. * map: This is an all-purpose remapping filter. Best to read the man page to see what this does. (These filters are not really related to each other, except that the truncate filter turned out to be necessary to test the map filter.) Rich.
Richard W.M. Jones
2018-Jul-31 19:55 UTC
[Libguestfs] [PATCH nbdkit 1/4] Add truncate filter for truncating or extending the size of plugins.
This can truncate, extend, or round up/down to a multiple.
---
common-rules.mk | 3 +-
configure.ac | 1 +
filters/offset/nbdkit-offset-filter.pod | 7 +-
filters/partition/nbdkit-partition-filter.pod | 1 +
filters/truncate/Makefile.am | 60 ++++
filters/truncate/nbdkit-truncate-filter.pod | 87 ++++++
filters/truncate/truncate.c | 261 ++++++++++++++++++
7 files changed, 417 insertions(+), 3 deletions(-)
diff --git a/common-rules.mk b/common-rules.mk
index 01beff5..f600293 100644
--- a/common-rules.mk
+++ b/common-rules.mk
@@ -68,7 +68,8 @@ filters = \
log \
nozero \
offset \
- partition
+ partition \
+ truncate
plugindir = $(libdir)/nbdkit/plugins
filterdir = $(libdir)/nbdkit/filters
diff --git a/configure.ac b/configure.ac
index b87efb9..e8d0a38 100644
--- a/configure.ac
+++ b/configure.ac
@@ -578,6 +578,7 @@ AC_CONFIG_FILES([Makefile
filters/nozero/Makefile
filters/offset/Makefile
filters/partition/Makefile
+ filters/truncate/Makefile
src/Makefile
src/nbdkit.pc
tests/Makefile])
diff --git a/filters/offset/nbdkit-offset-filter.pod
b/filters/offset/nbdkit-offset-filter.pod
index ee8061b..6d8f9be 100644
--- a/filters/offset/nbdkit-offset-filter.pod
+++ b/filters/offset/nbdkit-offset-filter.pod
@@ -32,7 +32,9 @@ file/device.
=back
Note it is an error if the offset and/or range specify data which lies
-beyond the end of the underlying device.
+beyond the end of the underlying device. Use
+L<nbdkit-truncate-filter(1)> to truncate or extend the size of
+plugins.
=head1 EXAMPLES
@@ -65,7 +67,8 @@ You can then serve the partition only using:
L<nbdkit(1)>,
L<nbdkit-file-plugin(1)>,
L<nbdkit-filter(3)>,
-L<nbdkit-partition-filter(1)>.
+L<nbdkit-partition-filter(1)>,
+L<nbdkit-truncate-filter(1)>.
=head1 AUTHORS
diff --git a/filters/partition/nbdkit-partition-filter.pod
b/filters/partition/nbdkit-partition-filter.pod
index bc5f346..71a7a3a 100644
--- a/filters/partition/nbdkit-partition-filter.pod
+++ b/filters/partition/nbdkit-partition-filter.pod
@@ -46,6 +46,7 @@ L<nbdkit(1)>,
L<nbdkit-file-plugin(1)>,
L<nbdkit-filter(3)>,
L<nbdkit-offset-filter(1)>,
+L<nbdkit-truncate-filter(1)>,
L<parted(8)>.
=head1 AUTHORS
diff --git a/filters/truncate/Makefile.am b/filters/truncate/Makefile.am
new file mode 100644
index 0000000..04ed4de
--- /dev/null
+++ b/filters/truncate/Makefile.am
@@ -0,0 +1,60 @@
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = nbdkit-truncate-filter.pod
+
+filter_LTLIBRARIES = nbdkit-truncate-filter.la
+
+nbdkit_truncate_filter_la_SOURCES = \
+ truncate.c \
+ $(top_srcdir)/include/nbdkit-filter.h
+
+nbdkit_truncate_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include
+nbdkit_truncate_filter_la_CFLAGS = \
+ $(WARNINGS_CFLAGS)
+nbdkit_truncate_filter_la_LDFLAGS = \
+ -module -avoid-version -shared
+
+if HAVE_POD
+
+man_MANS = nbdkit-truncate-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-truncate-filter.1: nbdkit-truncate-filter.pod
+ $(PODWRAPPER) --section=1 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
diff --git a/filters/truncate/nbdkit-truncate-filter.pod
b/filters/truncate/nbdkit-truncate-filter.pod
new file mode 100644
index 0000000..d52a440
--- /dev/null
+++ b/filters/truncate/nbdkit-truncate-filter.pod
@@ -0,0 +1,87 @@
+=head1 NAME
+
+nbdkit-truncate-filter - change the size of plugins
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=truncate plugin [truncate=SIZE]
+ [round-up=N] [round-down=N]
+
+=head1 DESCRIPTION
+
+C<nbdkit-truncate-filter> is a filter that changes the size of
+the underlying plugin. It can:
+
+=over 4
+
+=item *
+
+Make the plugin smaller (truncate it). Use the C<truncate=SIZE>
+parameter to set the smaller size.
+
+=item *
+
+Make the plugin larger (the additional bytes read back as zeroes).
+Use C<truncate=SIZE> to set the larger size.
+
+=item *
+
+Round the size of the plugin up or down to the next multiple of C<N>.
+Use either C<round-up=N> or C<round-down=N>.
+
+=back
+
+A common use for this filter is to handle NBD clients which have a
+problem dealing with device sizes which are not a multiple of 512
+bytes. Use C<round-up=512> to round the size up to the next multiple
+of 512 bytes. If the size is already a multiple of 512 bytes then
+this has no effect.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<truncate=SIZE>
+
+Set the absolute size in bytes of the apparent device. This may be
+smaller or larger or the same as the underlying plugin.
+
+If the size of larger than the underlying plugin, reading the extra
+space returns zeroes, and any non-zero writes will return an error
+back to the client.
+
+This parameter is optional.
+
+=item B<round-up=N>
+
+Round the size up to the next multiple of C<N> bytes. If the size of
+the underlying plugin is already a multiple of C<N> bytes, this has no
+effect.
+
+This parameter is optional.
+
+=item B<round-down=N>
+
+Round the size down to a multiple of C<N> bytes. If the size of the
+underlying plugin is already a multiple of C<N> bytes, this has no
+effect.
+
+This parameter is optional.
+
+=back
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-file-plugin(1)>,
+L<nbdkit-filter(3)>,
+L<nbdkit-offset-filter(1)>,
+L<nbdkit-partition-filter(1)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2018 Red Hat Inc.
diff --git a/filters/truncate/truncate.c b/filters/truncate/truncate.c
new file mode 100644
index 0000000..eccf8cd
--- /dev/null
+++ b/filters/truncate/truncate.c
@@ -0,0 +1,261 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <errno.h>
+
+#include <nbdkit-filter.h>
+
+#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
+
+/* These are the parameters. */
+static int64_t truncate = -1, round_up = -1, round_down = -1;
+
+/* The real size of the underlying plugin. */
+static int64_t real_size;
+
+/* The calculated size after applying the parameters. */
+static int64_t size;
+
+/* Called for each key=value passed on the command line. */
+static int
+truncate_config (nbdkit_next_config *next, void *nxdata,
+ const char *key, const char *value)
+{
+ if (strcmp (key, "truncate") == 0) {
+ truncate = nbdkit_parse_size (value);
+ if (truncate == -1)
+ return -1;
+ return 0;
+ }
+ else if (strcmp (key, "round-up") == 0) {
+ round_up = nbdkit_parse_size (value);
+ if (round_up == -1)
+ return -1;
+ if (round_up == 0) {
+ nbdkit_error ("if set, the round-up parameter must be > 0");
+ return -1;
+ }
+ return 0;
+ }
+ else if (strcmp (key, "round-down") == 0) {
+ round_down = nbdkit_parse_size (value);
+ if (round_down == -1)
+ return -1;
+ if (round_down == 0) {
+ nbdkit_error ("if set, the round-down parameter must be >
0");
+ return -1;
+ }
+ return 0;
+ }
+ else
+ return next (nxdata, key, value);
+}
+
+#define truncate_config_help \
+ "truncate=<SIZE> The new size.\n" \
+ "round-up=<N> Round up to next multiple of
N.\n" \
+ "round-down=<N> Round down to multiple of N."
+
+static int64_t truncate_get_size (struct nbdkit_next_ops *next_ops, void
*nxdata, void *handle);
+
+/* In prepare, force a call to get_size which sets the real_size & size
+ * globals.
+ */
+static int
+truncate_prepare (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle)
+{
+ int64_t r;
+
+ r = truncate_get_size (next_ops, nxdata, handle);
+ return r >= 0 ? 0 : -1;
+}
+
+/* Get the size. As a side effect, calculate the size to serve. */
+static int64_t
+truncate_get_size (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle)
+{
+ real_size = size = next_ops->get_size (nxdata);
+
+ /* The truncate, round-up and round-down parameters are treated as
+ * separate operations. It's possible to specify more than one,
+ * although perhaps not very useful.
+ */
+ if (truncate >= 0)
+ size = truncate;
+ if (round_up > 0)
+ size = (size + round_up - 1) & ~(round_up-1);
+ if (round_down > 0)
+ size &= ~(round_down-1);
+ return size;
+}
+
+/* Read data. */
+static int
+truncate_pread (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags, int *err)
+{
+ int r;
+ uint32_t n;
+
+ if (offset < real_size) {
+ if (offset + count <= real_size)
+ n = count;
+ else
+ n = real_size - offset;
+ r = next_ops->pread (nxdata, buf, n, offset, flags, err);
+ if (r == -1)
+ return -1;
+ count -= n;
+ buf += n;
+ }
+
+ if (count > 0)
+ memset (buf, 0, count);
+
+ return 0;
+}
+
+/* Return true iff the buffer is all zero bytes.
+ *
+ * The clever approach here was suggested by Eric Blake. See:
+ * https://www.redhat.com/archives/libguestfs/2017-April/msg00171.html
+ */
+static inline int
+is_zero (const char *buffer, size_t size)
+{
+ size_t i;
+ const size_t limit = size < 16 ? size : 16;
+
+ for (i = 0; i < limit; ++i)
+ if (buffer[i])
+ return 0;
+ if (size != limit)
+ return !memcmp (buffer, buffer + 16, size - 16);
+
+ return 1;
+}
+
+/* Write data. */
+static int
+truncate_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle,
+ const void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags, int *err)
+{
+ int r;
+ uint32_t n;
+
+ if (offset < real_size) {
+ if (offset + count <= real_size)
+ n = count;
+ else
+ n = real_size - offset;
+ r = next_ops->pwrite (nxdata, buf, n, offset, flags, err);
+ if (r == -1)
+ return -1;
+ count -= n;
+ buf += n;
+ }
+
+ if (count > 0) {
+ /* The caller must be writing zeroes, else it's an error. */
+ if (!is_zero (buf, count)) {
+ nbdkit_error ("truncate: write beyond end of underlying
device");
+ *err = EIO;
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+/* Trim data. */
+static int
+truncate_trim (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offset,
+ uint32_t flags, int *err)
+{
+ uint32_t n;
+
+ if (offset < real_size) {
+ if (offset + count <= real_size)
+ n = count;
+ else
+ n = real_size - offset;
+ return next_ops->trim (nxdata, n, offset, flags, err);
+ }
+ return 0;
+}
+
+/* Zero data. */
+static int
+truncate_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offset,
+ uint32_t flags, int *err)
+{
+ uint32_t n;
+
+ if (offset < real_size) {
+ if (offset + count <= real_size)
+ n = count;
+ else
+ n = real_size - offset;
+ return next_ops->zero (nxdata, n, offset, flags, err);
+ }
+ return 0;
+}
+
+static struct nbdkit_filter filter = {
+ .name = "truncate",
+ .longname = "nbdkit truncate filter",
+ .version = PACKAGE_VERSION,
+ .config = truncate_config,
+ .config_help = truncate_config_help,
+ .prepare = truncate_prepare,
+ .get_size = truncate_get_size,
+ .pread = truncate_pread,
+ .pwrite = truncate_pwrite,
+ .trim = truncate_trim,
+ .zero = truncate_zero,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
--
2.18.0
Richard W.M. Jones
2018-Jul-31 19:55 UTC
[Libguestfs] [PATCH nbdkit 2/4] tests: truncate: Add two simple tests of the truncate filter.
---
tests/Makefile.am | 7 +++
tests/test-truncate1.sh | 120 ++++++++++++++++++++++++++++++++++++++++
tests/test-truncate2.sh | 120 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 247 insertions(+)
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 12b7aae..4c602d7 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -79,6 +79,8 @@ EXTRA_DIST = \
test-random-sock.sh \
test-tls.sh \
test-tls-psk.sh \
+ test-truncate1.sh \
+ test-truncate2.sh \
test-vddk.sh \
test-version.sh \
test-version-filter.sh \
@@ -571,4 +573,9 @@ test_partition_SOURCES = test-partition.c test.h
test_partition_CFLAGS = $(WARNINGS_CFLAGS) $(LIBGUESTFS_CFLAGS)
test_partition_LDADD = libtest.la $(LIBGUESTFS_LIBS)
+# truncate filter tests.
+TESTS += \
+ test-truncate1.sh \
+ test-truncate2.sh
+
endif HAVE_PLUGINS
diff --git a/tests/test-truncate1.sh b/tests/test-truncate1.sh
new file mode 100755
index 0000000..61d2a50
--- /dev/null
+++ b/tests/test-truncate1.sh
@@ -0,0 +1,120 @@
+#!/bin/bash -
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Test the truncate filter using the pattern plugin.
+
+set -e
+set -x
+
+files="truncate1.out truncate1.pid truncate1.sock"
+rm -f $files
+
+# Test that qemu-io works
+if ! qemu-io --help >/dev/null; then
+ echo "$0: missing or broken qemu-io"
+ exit 77
+fi
+
+# Run nbdkit with pattern plugin and truncate filter in front.
+nbdkit -P truncate1.pid -U truncate1.sock \
+ --filter=truncate \
+ pattern size=503 \
+ truncate=512
+
+# We may have to wait a short time for the pid file to appear.
+for i in `seq 1 10`; do
+ if test -f truncate1.pid; then
+ break
+ fi
+ sleep 1
+done
+if ! test -f truncate1.pid; then
+ echo "$0: PID file was not created"
+ exit 1
+fi
+
+pid="$(cat truncate1.pid)"
+
+# Kill the nbdkit process on exit.
+cleanup ()
+{
+ status=$?
+
+ kill $pid
+ rm -f $files
+
+ exit $status
+}
+trap cleanup INT QUIT TERM EXIT ERR
+
+qemu-io -r -f raw 'nbd+unix://?socket=truncate1.sock' \
+ -c 'r -v 0 512' | grep -E '^[[:xdigit:]]+:' >
truncate1.out
+if [ "$(cat truncate1.out)" != "00000000: 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 08 ................
+00000010: 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 18 ................
+00000020: 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 28 ................
+00000030: 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00 38 .......0.......8
+00000040: 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 48 ...............H
+00000050: 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 58 .......P.......X
+00000060: 00 00 00 00 00 00 00 60 00 00 00 00 00 00 00 68 ...............h
+00000070: 00 00 00 00 00 00 00 70 00 00 00 00 00 00 00 78 .......p.......x
+00000080: 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 88 ................
+00000090: 00 00 00 00 00 00 00 90 00 00 00 00 00 00 00 98 ................
+000000a0: 00 00 00 00 00 00 00 a0 00 00 00 00 00 00 00 a8 ................
+000000b0: 00 00 00 00 00 00 00 b0 00 00 00 00 00 00 00 b8 ................
+000000c0: 00 00 00 00 00 00 00 c0 00 00 00 00 00 00 00 c8 ................
+000000d0: 00 00 00 00 00 00 00 d0 00 00 00 00 00 00 00 d8 ................
+000000e0: 00 00 00 00 00 00 00 e0 00 00 00 00 00 00 00 e8 ................
+000000f0: 00 00 00 00 00 00 00 f0 00 00 00 00 00 00 00 f8 ................
+00000100: 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 08 ................
+00000110: 00 00 00 00 00 00 01 10 00 00 00 00 00 00 01 18 ................
+00000120: 00 00 00 00 00 00 01 20 00 00 00 00 00 00 01 28 ................
+00000130: 00 00 00 00 00 00 01 30 00 00 00 00 00 00 01 38 .......0.......8
+00000140: 00 00 00 00 00 00 01 40 00 00 00 00 00 00 01 48 ...............H
+00000150: 00 00 00 00 00 00 01 50 00 00 00 00 00 00 01 58 .......P.......X
+00000160: 00 00 00 00 00 00 01 60 00 00 00 00 00 00 01 68 ...............h
+00000170: 00 00 00 00 00 00 01 70 00 00 00 00 00 00 01 78 .......p.......x
+00000180: 00 00 00 00 00 00 01 80 00 00 00 00 00 00 01 88 ................
+00000190: 00 00 00 00 00 00 01 90 00 00 00 00 00 00 01 98 ................
+000001a0: 00 00 00 00 00 00 01 a0 00 00 00 00 00 00 01 a8 ................
+000001b0: 00 00 00 00 00 00 01 b0 00 00 00 00 00 00 01 b8 ................
+000001c0: 00 00 00 00 00 00 01 c0 00 00 00 00 00 00 01 c8 ................
+000001d0: 00 00 00 00 00 00 01 d0 00 00 00 00 00 00 01 d8 ................
+000001e0: 00 00 00 00 00 00 01 e0 00 00 00 00 00 00 01 e8 ................
+000001f0: 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00
................" ]
+then
+ echo "$0: unexpected pattern:"
+ cat truncate1.out
+ exit 1
+fi
+
+# The cleanup() function is called implicitly on exit.
diff --git a/tests/test-truncate2.sh b/tests/test-truncate2.sh
new file mode 100755
index 0000000..5afef16
--- /dev/null
+++ b/tests/test-truncate2.sh
@@ -0,0 +1,120 @@
+#!/bin/bash -
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Test the truncate filter using the pattern plugin.
+
+set -e
+set -x
+
+files="truncate2.out truncate2.pid truncate2.sock"
+rm -f $files
+
+# Test that qemu-io works
+if ! qemu-io --help >/dev/null; then
+ echo "$0: missing or broken qemu-io"
+ exit 77
+fi
+
+# Run nbdkit with pattern plugin and truncate filter in front.
+nbdkit -P truncate2.pid -U truncate2.sock \
+ --filter=truncate \
+ pattern size=503 \
+ round-up=512
+
+# We may have to wait a short time for the pid file to appear.
+for i in `seq 1 10`; do
+ if test -f truncate2.pid; then
+ break
+ fi
+ sleep 1
+done
+if ! test -f truncate2.pid; then
+ echo "$0: PID file was not created"
+ exit 1
+fi
+
+pid="$(cat truncate2.pid)"
+
+# Kill the nbdkit process on exit.
+cleanup ()
+{
+ status=$?
+
+ kill $pid
+ rm -f $files
+
+ exit $status
+}
+trap cleanup INT QUIT TERM EXIT ERR
+
+qemu-io -r -f raw 'nbd+unix://?socket=truncate2.sock' \
+ -c 'r -v 0 512' | grep -E '^[[:xdigit:]]+:' >
truncate2.out
+if [ "$(cat truncate2.out)" != "00000000: 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 08 ................
+00000010: 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 18 ................
+00000020: 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 28 ................
+00000030: 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00 38 .......0.......8
+00000040: 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 48 ...............H
+00000050: 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 58 .......P.......X
+00000060: 00 00 00 00 00 00 00 60 00 00 00 00 00 00 00 68 ...............h
+00000070: 00 00 00 00 00 00 00 70 00 00 00 00 00 00 00 78 .......p.......x
+00000080: 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 88 ................
+00000090: 00 00 00 00 00 00 00 90 00 00 00 00 00 00 00 98 ................
+000000a0: 00 00 00 00 00 00 00 a0 00 00 00 00 00 00 00 a8 ................
+000000b0: 00 00 00 00 00 00 00 b0 00 00 00 00 00 00 00 b8 ................
+000000c0: 00 00 00 00 00 00 00 c0 00 00 00 00 00 00 00 c8 ................
+000000d0: 00 00 00 00 00 00 00 d0 00 00 00 00 00 00 00 d8 ................
+000000e0: 00 00 00 00 00 00 00 e0 00 00 00 00 00 00 00 e8 ................
+000000f0: 00 00 00 00 00 00 00 f0 00 00 00 00 00 00 00 f8 ................
+00000100: 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 08 ................
+00000110: 00 00 00 00 00 00 01 10 00 00 00 00 00 00 01 18 ................
+00000120: 00 00 00 00 00 00 01 20 00 00 00 00 00 00 01 28 ................
+00000130: 00 00 00 00 00 00 01 30 00 00 00 00 00 00 01 38 .......0.......8
+00000140: 00 00 00 00 00 00 01 40 00 00 00 00 00 00 01 48 ...............H
+00000150: 00 00 00 00 00 00 01 50 00 00 00 00 00 00 01 58 .......P.......X
+00000160: 00 00 00 00 00 00 01 60 00 00 00 00 00 00 01 68 ...............h
+00000170: 00 00 00 00 00 00 01 70 00 00 00 00 00 00 01 78 .......p.......x
+00000180: 00 00 00 00 00 00 01 80 00 00 00 00 00 00 01 88 ................
+00000190: 00 00 00 00 00 00 01 90 00 00 00 00 00 00 01 98 ................
+000001a0: 00 00 00 00 00 00 01 a0 00 00 00 00 00 00 01 a8 ................
+000001b0: 00 00 00 00 00 00 01 b0 00 00 00 00 00 00 01 b8 ................
+000001c0: 00 00 00 00 00 00 01 c0 00 00 00 00 00 00 01 c8 ................
+000001d0: 00 00 00 00 00 00 01 d0 00 00 00 00 00 00 01 d8 ................
+000001e0: 00 00 00 00 00 00 01 e0 00 00 00 00 00 00 01 e8 ................
+000001f0: 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00
................" ]
+then
+ echo "$0: unexpected pattern:"
+ cat truncate2.out
+ exit 1
+fi
+
+# The cleanup() function is called implicitly on exit.
--
2.18.0
Serve an arbitrary map of regions of the underlying plugin.
---
common-rules.mk | 1 +
configure.ac | 1 +
filters/map/Makefile.am | 61 +++
filters/map/map.c | 256 +++++++++
filters/map/maptype.c | 493 ++++++++++++++++++
filters/map/maptype.h | 76 +++
filters/map/nbdkit-map-filter.pod | 173 ++++++
filters/offset/nbdkit-offset-filter.pod | 1 +
filters/partition/nbdkit-partition-filter.pod | 1 +
plugins/pattern/nbdkit-pattern-plugin.pod | 1 +
tests/Makefile.am | 4 +
tests/test-map-empty.sh | 85 +++
12 files changed, 1153 insertions(+)
diff --git a/common-rules.mk b/common-rules.mk
index f600293..ae8d701 100644
--- a/common-rules.mk
+++ b/common-rules.mk
@@ -66,6 +66,7 @@ filters = \
delay \
fua \
log \
+ map \
nozero \
offset \
partition \
diff --git a/configure.ac b/configure.ac
index e8d0a38..2f40984 100644
--- a/configure.ac
+++ b/configure.ac
@@ -575,6 +575,7 @@ AC_CONFIG_FILES([Makefile
filters/delay/Makefile
filters/fua/Makefile
filters/log/Makefile
+ filters/map/Makefile
filters/nozero/Makefile
filters/offset/Makefile
filters/partition/Makefile
diff --git a/filters/map/Makefile.am b/filters/map/Makefile.am
new file mode 100644
index 0000000..96b5be8
--- /dev/null
+++ b/filters/map/Makefile.am
@@ -0,0 +1,61 @@
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = nbdkit-map-filter.pod
+
+filter_LTLIBRARIES = nbdkit-map-filter.la
+
+nbdkit_map_filter_la_SOURCES = \
+ map.c \
+ maptype.c \
+ maptype.h \
+ $(top_srcdir)/include/nbdkit-filter.h
+nbdkit_map_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include
+nbdkit_map_filter_la_CFLAGS = \
+ $(WARNINGS_CFLAGS)
+nbdkit_map_filter_la_LDFLAGS = \
+ -module -avoid-version -shared
+
+if HAVE_POD
+
+man_MANS = nbdkit-map-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-map-filter.1: nbdkit-map-filter.pod
+ $(PODWRAPPER) --section=1 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
diff --git a/filters/map/map.c b/filters/map/map.c
new file mode 100644
index 0000000..d551104
--- /dev/null
+++ b/filters/map/map.c
@@ -0,0 +1,256 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <inttypes.h>
+#include <ctype.h>
+#include <assert.h>
+
+#include <nbdkit-filter.h>
+
+#include "maptype.h"
+
+#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
+
+static char *filename; /* Map filename. */
+
+static void
+map_unload (void)
+{
+ free (filename);
+}
+
+/* Expect map=filename on the command line, pass everything else
+ * through.
+ */
+static int
+map_config (nbdkit_next_config *next, void *nxdata,
+ const char *key, const char *value)
+{
+ if (strcmp (key, "map") == 0) {
+ filename = nbdkit_realpath (value);
+ if (filename == NULL)
+ return -1;
+ return 0;
+ }
+ else
+ return next (nxdata, key, value);
+}
+
+/* Check that map parameter was supplied. */
+static int
+map_config_complete (nbdkit_next_config_complete *next, void *nxdata)
+{
+ if (filename == NULL) {
+ nbdkit_error ("map=<filename> must be passed to the map
filter");
+ return -1;
+ }
+
+ return next (nxdata);
+}
+
+#define map_config_help \
+ "map=<FILENAME> (required) Map file."
+
+struct handle {
+ /* We have to load the map file separately for each handle
+ * for a couple of reasons, the second one being critical:
+ *
+ * (1) The map file might change.
+ *
+ * (2) The size of the underlying plugin affects the behaviour
+ * of open-ended intervals in the map.
+ */
+ struct map map;
+};
+
+static void *
+map_open (nbdkit_next_open *next, void *nxdata, int readonly)
+{
+ struct handle *h;
+
+ if (next (nxdata, readonly) == -1)
+ return NULL;
+
+ h = malloc (sizeof *h);
+ if (h == NULL) {
+ nbdkit_error ("malloc: %m");
+ return NULL;
+ }
+ map_init (&h->map);
+
+ return h;
+}
+
+/* Force an early call to get the size of the map, then read the
+ * map file.
+ */
+static int
+map_prepare (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle)
+{
+ struct handle *h = handle;
+ int64_t size;
+
+ size = next_ops->get_size (nxdata);
+ if (size == -1)
+ return -1;
+ nbdkit_debug ("map: plugin size: %" PRIi64, size);
+
+ if (map_load_from_file (filename, size, &h->map) == -1)
+ return -1;
+
+ return 0;
+}
+
+static void
+map_close (void *handle)
+{
+ struct handle *h = handle;
+
+ map_free (&h->map);
+ free (h);
+}
+
+/* Get size. */
+static int64_t
+map_get_size (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle)
+{
+ struct handle *h = handle;
+ int64_t r;
+
+ r = map_size (&h->map);
+ nbdkit_debug ("map: filter size: %" PRIi64, r);
+ return r;
+}
+
+/* Read data. */
+struct pread_data {
+ struct nbdkit_next_ops *next_ops;
+ void *nxdata;
+ void *buf;
+ uint32_t flags;
+ int *err;
+};
+
+static int
+do_pread (void *vp, uint32_t count, uint64_t offs)
+{
+ struct pread_data *data = vp;
+
+ if (data->next_ops->pread (data->nxdata, data->buf,
+ count, offs, data->flags, data->err) == -1)
+ return -1;
+ data->buf += count;
+ return 0;
+}
+
+static int
+do_pread_unmapped (void *vp, uint32_t count)
+{
+ struct pread_data *data = vp;
+
+ /* Unmapped data reads as zeroes. */
+ memset (data->buf, 0, count);
+ return 0;
+}
+
+static int
+map_pread (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, void *buf, uint32_t count, uint64_t offs,
+ uint32_t flags, int *err)
+{
+ struct handle *h = handle;
+ struct pread_data data = {
+ .next_ops = next_ops,
+ .nxdata = nxdata,
+ .buf = buf,
+ .flags = flags,
+ .err = err,
+ };
+
+ return map_iter (&h->map, count, offs, &data, do_pread,
do_pread_unmapped);
+}
+
+/* Write data. */
+static int
+map_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle,
+ const void *buf, uint32_t count, uint64_t offs, uint32_t flags,
+ int *err)
+{
+ abort ();
+}
+
+/* Trim data. */
+static int
+map_trim (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offs, uint32_t flags,
+ int *err)
+{
+ abort ();
+}
+
+/* Zero data. */
+static int
+map_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offs, uint32_t flags,
+ int *err)
+{
+ abort ();
+}
+
+static struct nbdkit_filter filter = {
+ .name = "map",
+ .longname = "nbdkit map filter",
+ .version = PACKAGE_VERSION,
+ .unload = map_unload,
+ .config = map_config,
+ .config_complete = map_config_complete,
+ .config_help = map_config_help,
+ .open = map_open,
+ .prepare = map_prepare,
+ .close = map_close,
+ .get_size = map_get_size,
+ .pread = map_pread,
+ .pwrite = map_pwrite,
+ .trim = map_trim,
+ .zero = map_zero,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
diff --git a/filters/map/maptype.c b/filters/map/maptype.c
new file mode 100644
index 0000000..ef99363
--- /dev/null
+++ b/filters/map/maptype.c
@@ -0,0 +1,493 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <assert.h>
+
+#include <nbdkit-filter.h>
+
+#include "maptype.h"
+
+/* Produce additional debugging of this module. Only useful for
+ * finding bugs in the module, so this should normally be disabled.
+ */
+#define MAPTYPE_DEBUG 0
+
+/* Notes on the implementation.
+ *
+ * Throughout the filter we use the following terminology:
+ *
+ * request / requested etc: The client requested range of bytes to
+ * read or update.
+ *
+ * plugin: The target after the client request is mapped. This is
+ * what is passed along to the underlying plugin (or next filter in
+ * the chain).
+ *
+ * mappings: Single entries (lines) in the map file. They are of the
+ * form (plugin, request), ie. the mapping is done backwards.
+ *
+ * interval: start-end or (start, length).
+ *
+ * Only one mapping can apply to each requested byte. This fact is
+ * crucial as it allows us to store the mappings in a simple array
+ * with no overlapping intervals, and use an efficient binary search
+ * to map incoming requests to the plugin.
+ *
+ * When we read the map file we start with an empty array and add the
+ * intervals to it. At all times we must maintain the invariant that
+ * no intervals in the array may overlap, and therefore we have to
+ * split existing intervals as required. Earlier mappings are
+ * discarded where they overlap with later mappings.
+ */
+
+/* Compare entries by rq_start. */
+static int
+mapping_compare (const void *mv1, const void *mv2)
+{
+ const struct mapping *m1 = mv1;
+ const struct mapping *m2 = mv2;
+
+ if (m1->rq_start < m2->rq_start)
+ return -1;
+ else if (m1->rq_start > m2->rq_start)
+ return 1;
+ else
+ return 0;
+}
+
+/* Return true if two mappings overlap in the request range. */
+static int
+mappings_overlap (const struct mapping *m1, const struct mapping *m2)
+{
+ return m1->rq_end >= m2->rq_start && m1->rq_start <=
m2->rq_end;
+}
+
+void
+map_init (struct map *map)
+{
+ map->nr_map = 0;
+ map->map = NULL;
+}
+
+void
+map_free (struct map *map)
+{
+ if (!map) return;
+ free (map->map);
+ map_init (map);
+}
+
+/* Return the highest address in the map+1. */
+int64_t
+map_size (struct map *map)
+{
+ if (map->nr_map == 0)
+ return 0;
+ else
+ return map->map[map->nr_map-1].rq_end + 1;
+}
+
+/* Add a single new mapping at the end of the map. Does NOT maintain
+ * the invariant, use insert_mapping instead.
+ */
+static int
+add_mapping (struct map *map, const struct mapping *mapping)
+{
+ struct mapping *new_map;
+
+ map->nr_map++;
+ new_map = realloc (map->map, map->nr_map * sizeof map->map[0]);
+ if (new_map == NULL) {
+ nbdkit_error ("realloc: %m");
+ return -1;
+ }
+ map->map = new_map;
+ map->map[map->nr_map-1] = *mapping;
+ return 0;
+}
+
+/* Insert a single new mapping into the map. By splitting
+ * and discarding intervals, this maintains the invariant
+ * described above.
+ */
+static int
+insert_mapping (struct map *map, const struct mapping *new_mapping)
+{
+ size_t i;
+
+ /* Adjust existing mappings if they overlap with this mapping. */
+ for (i = 0; i < map->nr_map; ++i) {
+ if (mappings_overlap (&map->map[i], new_mapping)) {
+ /* The four cases are:
+ *
+ * existing +---+
+ * new +-------------------+
+ * => erase existing mapping
+ *
+ * existing +-------------------+
+ * new +---+
+ * => split existing mapping into two
+ *
+ * existing +-----------+
+ * new +-----+
+ * => adjust start of existing mapping
+ *
+ * existing +-----------+
+ * new +-----+
+ * => adjust end of existing mapping
+ */
+ if (map->map[i].rq_start >= new_mapping->rq_start &&
+ map->map[i].rq_end <= new_mapping->rq_end) {
+ /* Erase map[i]. */
+ memmove (&map->map[i], &map->map[i+1],
+ (map->nr_map-i-1) * sizeof map->map[0]);
+ map->nr_map--;
+ i--;
+ }
+ else if (map->map[i].rq_start < new_mapping->rq_start &&
+ map->map[i].rq_end > new_mapping->rq_end) {
+ struct mapping second;
+ uint64_t offset;
+
+ /* Split map[i] by reducing map[i] and creating second mapping. */
+ second.lineno = map->map[i].lineno;
+ second.rq_start = new_mapping->rq_end+1;
+ second.rq_end = map->map[i].rq_end;
+ offset = new_mapping->rq_end+1 - map->map[i].rq_start;
+ second.plugin_start = map->map[i].plugin_start + offset;
+ if (add_mapping (map, &second) == -1)
+ return -1;
+ map->map[i].rq_end = new_mapping->rq_start-1;
+ }
+ else if (map->map[i].rq_start >= new_mapping->rq_start) {
+ uint64_t offset;
+
+ /* Adjust start of map[i]. */
+ offset = new_mapping->rq_end+1 - map->map[i].rq_start;
+ map->map[i].rq_start = new_mapping->rq_end+1;
+ map->map[i].plugin_start += offset;
+ }
+ else if (map->map[i].rq_end <= new_mapping->rq_end)
+ /* Adjust end of map[i]. */
+ map->map[i].rq_end = new_mapping->rq_start-1;
+ else
+ abort (); /* Should never happen. */
+ }
+ }
+
+ /* Add new mapping at the end. Note that the new mapping does not
+ * need to be adjusted.
+ */
+ return add_mapping (map, new_mapping);
+}
+
+/* Load the map file. */
+int
+map_load_from_file (const char *filename, int64_t plugin_size,
+ struct map *map)
+{
+ /* Set of whitespace in the map file. */
+ static const char whitespace[] = " \t\n\r";
+
+ FILE *fp;
+ ssize_t r;
+ size_t len = 0;
+ char *line = NULL;
+ int lineno = 0;
+ size_t i;
+
+ fp = fopen (filename, "r");
+ if (fp == NULL) {
+ nbdkit_error ("open: %s: %m", filename);
+ return -1;
+ }
+ while ((r = getline (&line, &len, fp)) != -1) {
+ char *p, *q, *saveptr;
+ size_t n;
+ int64_t i;
+ int64_t length; /* signed because -1 means end of input */
+ struct mapping mapping;
+
+ lineno++;
+ mapping.lineno = lineno;
+
+ /* Remove anything after # (comment) character. */
+ p = strchr (line, '#');
+ if (p)
+ *p = '\0';
+
+ /* Trim whitespace at beginning of the line. */
+ n = strspn (line, whitespace);
+ if (n > 0)
+ memmove (line, &line[n], strlen (&line[n]));
+
+ /* Trim whitespace at end of the line (including \n and \r). */
+ n = strlen (line);
+ while (n > 0) {
+ if (strspn (&line[n-1], whitespace) == 0)
+ break;
+ line[n-1] = '\0';
+ n--;
+ }
+
+ /* Ignore blank lines. */
+ if (n == 0)
+ continue;
+
+ /* First field.
+ * Expecting: "start,length" or "start-end" or
"start-" or "start".
+ */
+ p = strtok_r (line, whitespace, &saveptr);
+ if (p == NULL) {
+ /* AFAIK this can never happen. */
+ nbdkit_error ("%s:%d: could not read token", filename, lineno);
+ goto err;
+ }
+ if ((q = strchr (p, ',')) != NULL) { /* start,length */
+ *q = '\0'; q++;
+ i = nbdkit_parse_size (p);
+ if (i == -1)
+ goto err;
+ mapping.plugin_start = i;
+ i = nbdkit_parse_size (q);
+ if (i == -1)
+ goto err;
+ length = i;
+ }
+ else if ((q = strchr (p, '-')) != NULL) { /* start-end or start- */
+ *q = '\0'; q++;
+ i = nbdkit_parse_size (p);
+ if (i == -1)
+ goto err;
+ mapping.plugin_start = i;
+ if (*q == '\0')
+ length = -1;
+ else {
+ i = nbdkit_parse_size (q);
+ if (i == -1)
+ goto err;
+ /* Note: 100-99 is allowed (means zero length). However the
+ * length must not be negative.
+ */
+ if (i < mapping.plugin_start-1) {
+ nbdkit_error ("%s:%d: length < 0", filename, lineno);
+ goto err;
+ }
+ length = i - mapping.plugin_start + 1;
+ }
+ }
+ else { /* start */
+ i = nbdkit_parse_size (p);
+ if (i == -1)
+ goto err;
+ mapping.plugin_start = i;
+ length = -1;
+ }
+
+ /* length == -1 means to the end of the plugin. Calculate that. */
+ if (length == -1)
+ length = plugin_size - mapping.plugin_start;
+
+ /* A zero-length mapping isn't an error, but can be ignored
immediately. */
+ if (length == 0)
+ continue;
+
+ /* Second field. Expecting a single offset. */
+ p = strtok_r (NULL, whitespace, &saveptr);
+ i = nbdkit_parse_size (p);
+ if (i == -1)
+ goto err;
+ mapping.rq_start = i;
+
+ /* Calculate the end of the output region. */
+ mapping.rq_end = mapping.rq_start + length - 1;
+
+ /* We just ignore everything on the line after the second field.
+ * But don't put anything there, we might use this for something
+ * in future.
+ */
+
+ /* Debug the line as it was read. */
+ nbdkit_debug ("map: %s:%d: "
+ "plugin.start=%" PRIu64 ",
plugin.length=%" PRIi64 ", "
+ "request.start=%" PRIu64 ",
request.end=%" PRIu64,
+ filename, lineno,
+ mapping.plugin_start, length,
+ mapping.rq_start, mapping.rq_end);
+
+ /* Insert into the map. */
+ if (insert_mapping (map, &mapping) == -1)
+ goto err;
+ }
+
+ fclose (fp);
+ free (line);
+
+ /* The map maintains an invariant that no intervals are overlapping.
+ * However it is not yet sorted which we need for efficient lookups
+ * (using bsearch), so do that now.
+ */
+ if (map->nr_map > 0)
+ qsort (map->map, map->nr_map, sizeof map->map[0],
mapping_compare);
+
+ /* Check there are no overlapping mappings. Because of the sort
+ * above we only need to check adjacent pairs so this is quite
+ * efficient and we can do it every time.
+ */
+ if (map->nr_map > 0)
+ for (i = 0; i < map->nr_map-1; ++i)
+ assert (!mappings_overlap (&map->map[i], &map->map[i+1]));
+
+ /* If debugging print the final map. */
+ for (i = 0; i < map->nr_map; ++i)
+ nbdkit_debug ("map: map[%zu] = [%" PRIu64 "-%" PRIu64
":"
+ "%" PRIu64 "] (from %s:%d)",
+ i, map->map[i].rq_start, map->map[i].rq_end,
+ map->map[i].plugin_start, filename,
map->map[i].lineno);
+
+ return 0;
+
+ err:
+ fclose (fp);
+ free (line);
+ map_free (map);
+ return -1;
+}
+
+/* Look up a single address in the map.
+ *
+ * If mapped, returns the mapping index (in map[]). In this case
+ * *is_mapped == true.
+ *
+ * If unmapped, returns the mapping index of the next mapped area
+ * (which can be >= nr_map if there are no more mappings). In this
+ * case *is_mapped == false.
+ *
+ * Note this only works because of the invariant that mappings are not
+ * allowed to overlap. See description at top of file.
+ */
+size_t
+map_lookup (const struct map *map, uint64_t p, int *is_mapped)
+{
+ size_t lo, hi, mid;
+
+ /* Deal with the special case where the map is completely empty
+ * because it makes the rest of the code easier.
+ */
+ if (map->nr_map == 0) {
+ *is_mapped = 0;
+ return 0;
+ }
+
+ /* Unmapped, before the first interval? */
+ if (p < map->map[0].rq_start) {
+ *is_mapped = 0;
+ return 0;
+ }
+
+ /* Do a binary search to find the mapping. */
+ lo = 0;
+ hi = map->nr_map;
+ while (lo < hi) {
+ mid = (lo + hi) / 2;
+ if (map->map[mid].rq_start <= p && p <=
map->map[mid].rq_end)
+ lo = hi = mid; /* terminates loop */
+ else if (p < map->map[mid].rq_start && hi != mid)
+ hi = mid;
+ else if (map->map[mid].rq_end < p && lo != mid)
+ lo = mid;
+ else
+ lo = hi = mid; /* terminates loop */
+ }
+
+ *is_mapped = p <= map->map[lo].rq_end;
+ return lo;
+}
+
+/* Iterate over the map. */
+int
+map_iter (const struct map *map,
+ uint32_t count, uint64_t offs, void *data,
+ int (*mapped_fn) (void *data, uint32_t count, uint64_t offs),
+ int (*unmapped_fn) (void *data, uint32_t count))
+{
+ size_t i;
+ int is_mapped;
+ size_t len;
+
+ while (count > 0) {
+ i = map_lookup (map, offs, &is_mapped);
+
+ if (MAPTYPE_DEBUG)
+ nbdkit_debug ("map: iter: "
+ "offset %" PRIu64 " %s map[%zu] = "
+ "[%" PRIu64 "-%" PRIu64 ":%"
PRIu64 "]",
+ offs, is_mapped ? "mapped to" : "unmapped
below", i,
+ map->map[i].rq_start, map->map[i].rq_end,
+ map->map[i].plugin_start);
+
+ if (is_mapped) {
+ len = map->map[i].rq_end - offs + 1;
+ if (mapped_fn (data,
+ len < count ? len : count,
+ map->map[i].plugin_start + offs
+ - map->map[i].rq_start) == -1)
+ return -1;
+ }
+ else {
+ if (i < map->nr_map)
+ len = map->map[i].rq_start - offs;
+ else
+ len = count; /* No more mappings above this one. */
+ if (unmapped_fn (data,
+ len < count ? len : count) == -1)
+ return -1;
+ }
+
+ if (len < count) {
+ offs += len;
+ count -= len;
+ }
+ else
+ count = 0; /* Fulfilled whole request. */
+ }
+
+ return 0;
+}
diff --git a/filters/map/maptype.h b/filters/map/maptype.h
new file mode 100644
index 0000000..b24e572
--- /dev/null
+++ b/filters/map/maptype.h
@@ -0,0 +1,76 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef NBDKIT_MAPTYPE_H
+#define NBDKIT_MAPTYPE_H
+
+struct mapping {
+ /* The start and end (inclusive) of the requested interval. */
+ uint64_t rq_start, rq_end;
+
+ /* The interval this maps to in the plugin. end/length is implied. */
+ uint64_t plugin_start;
+
+ /* The source line for the mapping. */
+ int lineno;
+};
+
+struct map {
+ struct mapping *map; /* List of mappings. */
+ size_t nr_map; /* Number of entries in map array. */
+};
+
+/* Initialize the map structure. */
+extern void map_init (struct map *map);
+
+/* This only frees the map->map array. */
+extern void map_free (struct map *map);
+
+/* Return the highest address in the map + 1. */
+extern int64_t map_size (struct map *map);
+
+/* Load the map from a file, constructing the map structure. */
+extern int map_load_from_file (const char *filename, int64_t plugin_size,
+ struct map *map);
+
+/* Lookup a single address in the map. */
+extern size_t map_lookup (const struct map *map, uint64_t p, int *is_mapped);
+
+/* Iterate over the map. */
+typedef int (*mapped_fn_t) (void *data, uint32_t count, uint64_t offs);
+typedef int (*unmapped_fn_t) (void *data, uint32_t count);
+extern int map_iter (const struct map *map,
+ uint32_t count, uint64_t offs, void *data,
+ mapped_fn_t mapped_fn, unmapped_fn_t unmapped_fn);
+
+#endif /* NBDKIT_MAPTYPE_H */
diff --git a/filters/map/nbdkit-map-filter.pod
b/filters/map/nbdkit-map-filter.pod
new file mode 100644
index 0000000..05c6b90
--- /dev/null
+++ b/filters/map/nbdkit-map-filter.pod
@@ -0,0 +1,173 @@
+=head1 NAME
+
+nbdkit-map-filter - nbdkit map filter
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=map plugin map=FILENAME [plugin-args...]
+
+=head1 DESCRIPTION
+
+C<nbdkit-map-filter> is a filter that can serve an arbitrary map of
+regions of the underlying plugin.
+
+It is driven by a map file that contains a list of regions from the
+plugin and where they should be served in the output.
+
+For example this map would divide the plugin data into two 16K halves
+and swap them over:
+
+ # map file
+ 0,16K 16K # aaaaa
+ 16K,16K 0 # bbbbb
+
+When visualised, this map file looks like:
+
+ ┌──────────────┬──────────────┬─── ─ ─ ─
+ Plugin serves ... │ aaaaaaaaaaaa │ bbbbbbbbbbbb │ (extra data)
+ │ 16K │ 16K │
+ └──────────────┴──────────────┴─── ─ ─ ─
+ │ │
+ Filter │ ┌─────────┘
+ transforms ... └──────────────┐
+ │ │
+ ┌──────────▼───┬─────▼────────┐
+ Client sees ... │ bbbbbbbbbbbb │ aaaaaaaaaaaa │
+ └──────────────┴──────────────┘
+
+This is how to simulate L<nbdkit-offset-filter(1)> C<offset> and
+C<range> parameters:
+
+ # offset,range
+ 1M,32M 0
+
+ ┌─────┬─────────────────────┬─── ─ ─ ─
+ Plugin serves ... │ │ ccccccccccccccccccc │ (extra data)
+ │ 1M │ 32M │
+ └─────┴─────────────────────┴─── ─ ─ ─
+ Filter │
+ transforms ... ┌─────┘
+ │
+ ┌─────────▼───────────┐
+ Client sees ... │ ccccccccccccccccccc │
+ └─────────────────────┘
+
+You can also do obscure things like duplicating regions of the source:
+
+ # map file
+ 0,16K 0
+ 0,16K 16K
+
+ ┌──────────────┬─── ─ ─ ─
+ Plugin serves ... │ aaaaaaaaaaaa │ (extra data)
+ │ 16K │
+ └──────────────┴─── ─ ─ ─
+ Filter │
+ transforms ... └───┬──────────┐
+ │ │
+ ┌─────────▼────┬─────▼────────┐
+ Client sees ... │ aaaaaaaaaaaa │ aaaaaaaaaaaa │
+ └──────────────┴──────────────┘
+
+=head2 Map file format
+
+The map file describes how regions from the plugin are mapped to the
+output. There is one line per mapping. Blank lines are ignored.
+C<#> indicates a comment.
+
+Each line (mapping) has one of the following forms:
+
+ start,length offset # see "start,length" below
+ start-end offset # see "start-end" below
+ start- offset # see "start to end of plugin" below
+ start offset # see "start to end of plugin" below
+
+=head2 C<start,length>
+
+ start,length offset
+
+means that the source region starting at byte C<start>, for
C<length>
+bytes, is mapped to C<offset> to C<offset+length-1> in the output.
+
+For example:
+
+ 16K,8K 0
+
+maps the 8K-sized region starting at 16K in the source to the
+beginning (ie. from offset 0) of the output.
+
+=head2 C<start-end>
+
+ start-end offset
+
+means that the source region starting at byte C<start> through to byte
+C<end> (inclusive) is mapped to C<offset> through to
+C<offset+(end-start)> in the output.
+
+For example:
+
+ 1024-2047 2048
+
+maps the region starting at byte 1024 and ending at byte 2047
+(inclusive) to bytes 2048-3071 in the output.
+
+=head2 C<start> to end of plugin
+
+ start- offset
+ start offset
+
+If the C<end> field is omitted it means "up to the end of the
+underlying plugin".
+
+=head2 Size modifiers
+
+You can use the usual power-of-2 size modifiers like C<K>, C<M>
etc.
+
+=head2 Overlapping mappings
+
+If there are multiple mappings in the map file that may apply to a
+particular byte of the filter output then it is the last one in the
+file which applies.
+
+=head2 Virtual size
+
+The virtual size of the filter output finishes at the last byte of the
+final mapped region. Note this is usually different from the size of
+the underlying plugin.
+
+=head2 Unmapped regions
+
+Any unmapped region (followed by a mapped region and therefore not
+beyond the virtual size) reads as zero and returns an error if
+written.
+
+Any mapping or part of a mapping where the source region refers beyond
+the end of the underlying plugin reads as zero and returns an error if
+written.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<map=FILENAME>
+
+Specify the map filename (required). See L</Map file format> above.
+
+=back
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-file-plugin(1)>,
+L<nbdkit-filter(3)>,
+L<nbdkit-offset-filter(1)>,
+L<nbdkit-partition-filter(1)>,
+L<nbdkit-truncate-filter(1)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2018 Red Hat Inc.
diff --git a/filters/offset/nbdkit-offset-filter.pod
b/filters/offset/nbdkit-offset-filter.pod
index 6d8f9be..96f9a13 100644
--- a/filters/offset/nbdkit-offset-filter.pod
+++ b/filters/offset/nbdkit-offset-filter.pod
@@ -67,6 +67,7 @@ You can then serve the partition only using:
L<nbdkit(1)>,
L<nbdkit-file-plugin(1)>,
L<nbdkit-filter(3)>,
+L<nbdkit-map-filter(1)>,
L<nbdkit-partition-filter(1)>,
L<nbdkit-truncate-filter(1)>.
diff --git a/filters/partition/nbdkit-partition-filter.pod
b/filters/partition/nbdkit-partition-filter.pod
index 71a7a3a..8d9aabb 100644
--- a/filters/partition/nbdkit-partition-filter.pod
+++ b/filters/partition/nbdkit-partition-filter.pod
@@ -45,6 +45,7 @@ image). To serve the first partition only use:
L<nbdkit(1)>,
L<nbdkit-file-plugin(1)>,
L<nbdkit-filter(3)>,
+L<nbdkit-map-filter(1)>,
L<nbdkit-offset-filter(1)>,
L<nbdkit-truncate-filter(1)>,
L<parted(8)>.
diff --git a/plugins/pattern/nbdkit-pattern-plugin.pod
b/plugins/pattern/nbdkit-pattern-plugin.pod
index d2bcd4d..d37d661 100644
--- a/plugins/pattern/nbdkit-pattern-plugin.pod
+++ b/plugins/pattern/nbdkit-pattern-plugin.pod
@@ -58,6 +58,7 @@ This parameter is required.
L<nbdkit(1)>,
L<nbdkit-plugin(3)>,
+L<nbdkit-map-filter(1)>,
L<nbdkit-null-plugin(1)>,
L<nbdkit-offset-filter(1)>,
L<nbdkit-random-plugin(1)>,
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 4c602d7..2306506 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -58,6 +58,7 @@ EXTRA_DIST = \
test-ip.sh \
test-log.sh \
test.lua \
+ test-map-empty.sh \
test-nozero.sh \
test_ocaml_plugin.ml \
test-ocaml.c \
@@ -547,6 +548,9 @@ TESTS += test-fua.sh
# log filter test.
TESTS += test-log.sh
+# map filter test.
+TESTS += test-map-empty.sh
+
# nozero filter test.
TESTS += test-nozero.sh
diff --git a/tests/test-map-empty.sh b/tests/test-map-empty.sh
new file mode 100755
index 0000000..3789234
--- /dev/null
+++ b/tests/test-map-empty.sh
@@ -0,0 +1,85 @@
+#!/bin/bash -
+# nbdkit
+# Copyright (C) 2018 Red Hat Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Test the map filter with an empty map file.
+
+set -e
+
+files="map-empty.out map-empty.pid map-empty.sock"
+rm -f $files
+
+# Test that qemu-img works
+if ! qemu-img --version >/dev/null; then
+ echo "$0: missing or broken qemu-img"
+ exit 77
+fi
+
+# Run nbdkit with pattern plugin and an empty map file on top.
+nbdkit -P map-empty.pid -U map-empty.sock \
+ --filter=map pattern size=10M map=/dev/null
+
+# We may have to wait a short time for the pid file to appear.
+for i in `seq 1 10`; do
+ if test -f map-empty.pid; then
+ break
+ fi
+ sleep 1
+done
+if ! test -f map-empty.pid; then
+ echo "$0: PID file was not created"
+ exit 1
+fi
+
+pid="$(cat map-empty.pid)"
+
+# Kill the nbdkit process on exit.
+cleanup ()
+{
+ status=$?
+
+ kill $pid
+ rm -f $files
+
+ exit $status
+}
+trap cleanup INT QUIT TERM EXIT ERR
+
+LANG=C qemu-img info 'nbd+unix://?socket=map-empty.sock' |
+ grep "^virtual size:" > map-empty.out
+if [ "$(cat map-empty.out)" != "virtual size: 0 (0 bytes)"
]; then
+ echo "$0: unexpected output:"
+ cat map-empty.out
+ exit 1
+fi
+
+# The cleanup() function is called implicitly on exit.
--
2.18.0
Richard W.M. Jones
2018-Jul-31 19:55 UTC
[Libguestfs] [PATCH nbdkit 4/4] tests: map: Add a stochastic test of the map filter.
---
.gitignore | 2 +
tests/Makefile.am | 5 +
tests/test-map-stochastic.c | 307 ++++++++++++++++++++++++++++++++++++
3 files changed, 314 insertions(+)
diff --git a/.gitignore b/.gitignore
index a84ad9a..da8b845 100644
--- a/.gitignore
+++ b/.gitignore
@@ -54,6 +54,7 @@ Makefile.in
/tests/ext2.img
/tests/file-data
/tests/keys.psk
+/tests/mapfile
/tests/offset-data
/tests/partition-disk
/tests/pki
@@ -67,6 +68,7 @@ Makefile.in
/tests/test-file
/tests/test-gzip
/tests/test-lua
+/tests/test-map-stochastic
/tests/test-memory
/tests/test-newstyle
/tests/test-nbd
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2306506..40efd73 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -550,6 +550,11 @@ TESTS += test-log.sh
# map filter test.
TESTS += test-map-empty.sh
+LIBGUESTFS_TESTS += test-map-stochastic
+
+test_map_stochastic_SOURCES = test-map-stochastic.c test.h
+test_map_stochastic_CFLAGS = $(WARNINGS_CFLAGS) $(LIBGUESTFS_CFLAGS)
+test_map_stochastic_LDADD = libtest.la $(LIBGUESTFS_LIBS)
# nozero filter test.
TESTS += test-nozero.sh
diff --git a/tests/test-map-stochastic.c b/tests/test-map-stochastic.c
new file mode 100644
index 0000000..10b37de
--- /dev/null
+++ b/tests/test-map-stochastic.c
@@ -0,0 +1,307 @@
+/* nbdkit
+ * Copyright (C) 2018 Red Hat Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS
IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+#include <time.h>
+
+#include <guestfs.h>
+
+#include "test.h"
+
+#define PATTERN_SIZE 65536 /* size in bytes of the input pattern */
+#define FILTER_SIZE 131072 /* size in bytes of the output of the filter */
+#define MAPFILE "mapfile" /* in tests directory */
+
+#define XSTR(s) STR(s)
+#define STR(s) #s
+
+static char pattern[PATTERN_SIZE]; /* underlying pattern data */
+static char expected[FILTER_SIZE]; /* after mapping */
+static char actual[FILTER_SIZE]; /* what we read from nbdkit --filter=map */
+
+static void create_pattern (void);
+static void create_map (void);
+static void mismatch_error (size_t output_size) __attribute__((noreturn));
+
+int
+main (int argc, char *argv[])
+{
+ guestfs_h *g;
+ int r;
+ size_t i, output_size;
+
+ srandom (time (NULL));
+
+ create_pattern ();
+
+ /* Create the random mapfile and associated expected data. */
+ create_map ();
+
+ /* We place a truncate filter in front of the map to round up its
+ * size to the next multiple of 512 bytes. Otherwise qemu chokes on
+ * the non-standard size of most randomly generated maps.
+ */
+ if (test_start_nbdkit ("-r",
+ "--filter", "truncate",
+ "--filter", "map",
+ "pattern", "size="
XSTR(PATTERN_SIZE),
+ "map=" MAPFILE,
+ "round-up=512",
+ NULL) == -1)
+ exit (EXIT_FAILURE);
+
+ g = guestfs_create ();
+ if (g == NULL) {
+ perror ("guestfs_create");
+ exit (EXIT_FAILURE);
+ }
+
+ r = guestfs_add_drive_opts (g, "",
+ GUESTFS_ADD_DRIVE_OPTS_READONLY, 1,
+ GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw",
+ GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "nbd",
+ GUESTFS_ADD_DRIVE_OPTS_SERVER, server,
+ -1);
+ if (r == -1)
+ exit (EXIT_FAILURE);
+
+ if (guestfs_launch (g) == -1)
+ exit (EXIT_FAILURE);
+
+ /* The size of the final device is not necessarily FILTER_SIZE. It
+ * might be smaller if there are no mappings that happen to cover
+ * the end. We can ask nbdkit for the size.
+ */
+ output_size = guestfs_blockdev_getsize64 (g, "/dev/sda");
+ if (output_size == -1)
+ exit (EXIT_FAILURE);
+
+ if (output_size > FILTER_SIZE) {
+ fprintf (stderr,
+ "test-map-stochastic: unexpected size returned for device:
"
+ "%zu > %d\n", output_size, FILTER_SIZE);
+ exit (EXIT_FAILURE);
+ }
+
+ i = 0;
+ while (i < output_size) {
+ /* Note that qemu will only issue reads on 512 byte boundaries
+ * so mostly this is pointless.
+ */
+ int n = random () % 2048;
+ size_t read;
+ char *buf;
+
+ if (n <= 0)
+ n = 1;
+ if (n > output_size-i)
+ n = output_size-i;
+
+ buf = guestfs_pread_device (g, "/dev/sda", n, (int64_t) i,
&read);
+ if (buf == NULL)
+ exit (EXIT_FAILURE);
+ memcpy (&actual[i], buf, read);
+ free (buf);
+
+ i += read;
+ }
+
+ if (memcmp (actual, expected, output_size) != 0)
+ mismatch_error (output_size);
+
+ unlink (MAPFILE);
+
+ guestfs_close (g);
+ exit (EXIT_SUCCESS);
+}
+
+/* Create the pattern data in the same way as the pattern plugin
+ * works. See nbdkit-pattern-plugin(1).
+ */
+static void
+create_pattern (void)
+{
+ size_t i;
+ uint64_t d;
+
+ for (i = 0; i < PATTERN_SIZE; i += 8) {
+ d = i;
+ d = htobe64 (d);
+ memcpy (&pattern[i], &d, 8);
+ }
+}
+
+/* Create the random map, write it to the map file, and map
+ * the pattern data accordingly.
+ */
+static const int rows[] = { 0, 1, 3, 5, 7, 11, 13, 17, 19, 23 };
+static const int alignments[] = { 1, 1, 1, 2, 4, 8, 512 };
+static const int lens[] = { 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 3, 3, 3,
+ 7, 8, 9,
+ 15, 16, 17,
+ 512, 1024, 2048, 8192,
+ 16384, 16384, 16384*2, 16384*2,
+ 16384*3, 16384*3, 16384*4, 16384*4,
+ 16384*5, 16384*6, 16384*7, 16384*8 };
+
+#define RANDOM_CHOICE(array) \
+ array[random () % (sizeof array / sizeof array[0])]
+
+static uint64_t
+random_alignment (uint64_t i)
+{
+ uint64_t alignment = RANDOM_CHOICE (alignments);
+
+ i &= ~(alignment - 1);
+ return i;
+}
+
+static void
+create_map (void)
+{
+ FILE *fp;
+ size_t i, j, nr_rows;
+ uint64_t start, length, to;
+ int end_form, open_end;
+
+ memset (expected, 0, sizeof expected);
+ memset (actual, 0, sizeof actual);
+
+ /* Pick a random number of mappings (rows in the map file). Can be 0. */
+ nr_rows = RANDOM_CHOICE (rows);
+
+ fp = fopen (MAPFILE, "w");
+ if (fp == NULL) {
+ perror (MAPFILE);
+ exit (EXIT_FAILURE);
+ }
+
+ fprintf (fp, "# %s for testing the map filter\n", MAPFILE);
+ fprintf (fp, "# generated randomly by %s\n", __FILE__);
+ fprintf (fp, "# pattern plugin (input) size: %d\n", PATTERN_SIZE);
+ fprintf (fp, "# map filter (max output) size: %d\n", FILTER_SIZE);
+ fprintf (fp, "# nr_rows: %zu\n", nr_rows);
+ fprintf (fp, "\n");
+
+ for (i = 0; i < nr_rows; ++i) {
+ /* Pick a random start point, length and destination (to). */
+ start = random () % PATTERN_SIZE;
+ start = random_alignment (start);
+ length = RANDOM_CHOICE (lens);
+ to = random () % FILTER_SIZE;
+ to = random_alignment (to);
+
+ /* Don't have a mapping going beyond the end of the output size
+ * we want.
+ */
+ if (to + length >= FILTER_SIZE)
+ length = FILTER_SIZE - to;
+
+ /* Choose randomly whether to use start-end or start,length. */
+ end_form = (random () % 2) == 1;
+
+ /* Randomly pick some open-ended mappings, but don't allow them if
+ * that would go beyond the output size we want.
+ */
+ open_end = 0;
+ if (PATTERN_SIZE - to <= FILTER_SIZE - start)
+ open_end = (random () % 8) == 1;
+
+ if (end_form) {
+ if (open_end)
+ fprintf (fp, "%" PRIu64 "-\t%" PRIu64
"\n", start, to);
+ else
+ fprintf (fp, "%" PRIu64 "-%" PRIu64 "\t%"
PRIu64 "\n",
+ start, start+length-1, to);
+ }
+ else {
+ if (open_end)
+ fprintf (fp, "%" PRIu64 "\t%" PRIu64
"\n", start, to);
+ else
+ fprintf (fp, "%" PRIu64 ",%" PRIu64 "\t%"
PRIu64 "\n",
+ start, length, to);
+ }
+
+ /* Perform the mapping into our expected[] array. */
+ for (j = 0; j < length; ++j) {
+ if (to+j <= FILTER_SIZE) {
+ if (start+j <= PATTERN_SIZE)
+ expected[to+j] = pattern[start+j];
+ else
+ expected[to+j] = 0; /* Bytes outside the source read as 0 */
+ }
+ }
+ }
+
+ fprintf (fp, "\n");
+ fprintf (fp, "# end of %s\n", MAPFILE);
+ fclose (fp);
+}
+
+static void
+dump_array (const char *array, size_t size)
+{
+ FILE *pp;
+
+ pp = popen ("hexdump -C", "w");
+ if (pp == NULL) {
+ perror ("popen: hexdump");
+ return;
+ }
+ fwrite (array, 1, size, pp);
+ pclose (pp);
+}
+
+static void
+mismatch_error (size_t output_size)
+{
+ fprintf (stderr, "test-map-stochastic: "
+ "actual data read back does not match expected data\n");
+ fprintf (stderr, "\n");
+ fprintf (stderr, "map contains:\n");
+ system ("cat " MAPFILE);
+ fprintf (stderr, "\n");
+ fprintf (stderr, "expected data:\n");
+ dump_array (expected, output_size);
+ fprintf (stderr, "\n");
+ fprintf (stderr, "actual data:\n");
+ dump_array (actual, output_size);
+ exit (EXIT_FAILURE);
+}
--
2.18.0
Richard W.M. Jones
2018-Jul-31 20:09 UTC
Re: [Libguestfs] [PATCH nbdkit 0/4] Add truncate and map filters.
On Tue, Jul 31, 2018 at 08:55:48PM +0100, Richard W.M. Jones wrote:> * map: This is an all-purpose remapping filter. Best to read the man > page to see what this does.I notice I didn't implement all the methods in the map filter :-( However still appreciate general comments/feedback on the approach. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Eric Blake
2018-Jul-31 21:48 UTC
Re: [Libguestfs] [PATCH nbdkit 1/4] Add truncate filter for truncating or extending the size of plugins.
On 07/31/2018 02:55 PM, Richard W.M. Jones wrote:> This can truncate, extend, or round up/down to a multiple. > --- > common-rules.mk | 3 +- > configure.ac | 1 + > filters/offset/nbdkit-offset-filter.pod | 7 +- > filters/partition/nbdkit-partition-filter.pod | 1 + > filters/truncate/Makefile.am | 60 ++++ > filters/truncate/nbdkit-truncate-filter.pod | 87 ++++++ > filters/truncate/truncate.c | 261 ++++++++++++++++++ > 7 files changed, 417 insertions(+), 3 deletions(-) >> +/* Get the size. As a side effect, calculate the size to serve. */ > +static int64_t > +truncate_get_size (struct nbdkit_next_ops *next_ops, void *nxdata, > + void *handle) > +{ > + real_size = size = next_ops->get_size (nxdata); > + > + /* The truncate, round-up and round-down parameters are treated as > + * separate operations. It's possible to specify more than one, > + * although perhaps not very useful. > + */ > + if (truncate >= 0) > + size = truncate; > + if (round_up > 0) > + size = (size + round_up - 1) & ~(round_up-1); > + if (round_down > 0) > + size &= ~(round_down-1);These last two operations presuppose that the user passed in a power of 2, but nothing actually enforces that the user did that. Something like 'round-up=5 round-down=3' thus produces strange results; I'd require powers of 2 to not have to worry about it elsewhere.> +/* Return true iff the buffer is all zero bytes. > + * > + * The clever approach here was suggested by Eric Blake. See: > + * https://www.redhat.com/archives/libguestfs/2017-April/msg00171.html > + */ > +static inline intWorth using bool (via <stdbool.h>) here...> +is_zero (const char *buffer, size_t size) > +{ > + size_t i; > + const size_t limit = size < 16 ? size : 16; > + > + for (i = 0; i < limit; ++i) > + if (buffer[i]) > + return 0;...and actually returning false/true, to match the documentation? Otherwise looks good to me. The offset filter duplicates behavior when real_size > truncated size, but the real win with this filter is when real_size < truncated size for quickly padding a plugin's actual length into a more practical length (qemu already does just that on every single image, rounding out to sector boundaries, although when it comes to writing in the last partial sector, qemu actually tries to resize the underlying file, for slightly less predictable outcomes than your nice trick of EIO unless the action doesn't change the all-zero read nature of the tail). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 07/31/2018 02:55 PM, Richard W.M. Jones wrote:> Serve an arbitrary map of regions of the underlying plugin. > ---> +#define map_config_help \ > + "map=<FILENAME> (required) Map file."Maybe worth a "see man page for format"? I have not closely read the code; at this point, it is just a quick review on the documentation/comments.> +/* Notes on the implementation. > + * > + * Throughout the filter we use the following terminology: > + * > + * request / requested etc: The client requested range of bytes to > + * read or update. > + * > + * plugin: The target after the client request is mapped. This is > + * what is passed along to the underlying plugin (or next filter in > + * the chain). > + * > + * mappings: Single entries (lines) in the map file. They are of the > + * form (plugin, request), ie. the mapping is done backwards. > + * > + * interval: start-end or (start, length). > + * > + * Only one mapping can apply to each requested byte. This fact isMaybe: Only the final applicable mapping can apply to each requested byte?> + * crucial as it allows us to store the mappings in a simple array > + * with no overlapping intervals, and use an efficient binary search > + * to map incoming requests to the plugin. > + * > + * When we read the map file we start with an empty array and add the > + * intervals to it. At all times we must maintain the invariant that > + * no intervals in the array may overlap, and therefore we have to > + * split existing intervals as required. Earlier mappings are > + * discarded where they overlap with later mappings.since this paragraph is all about maintaining the invariant in favor of each subsequent mapping, and thus the file format itself permits overlaps for shorthands for special-casing subregions. Is there a syntax for explicitly mentioning a subset is unmapped even after a larger mapping is applied first (perhaps useful for redacting a portion of a disk containing sensitive information)?> +static int > +insert_mapping (struct map *map, const struct mapping *new_mapping) > +{ > + size_t i; > + > + /* Adjust existing mappings if they overlap with this mapping. */ > + for (i = 0; i < map->nr_map; ++i) { > + if (mappings_overlap (&map->map[i], new_mapping)) { > + /* The four cases are: > + * > + * existing +---+ > + * new +-------------------+ > + * => erase existing mapping > + * > + * existing +-------------------+ > + * new +---+ > + * => split existing mapping into twoshould that be 'two/three'?> + * > + * existing +-----------+ > + * new +-----+ > + * => adjust start of existing mappingor is it really a case that you first split into two, then adjust one of the two> + * > + * existing +-----------+ > + * new +-----+ > + * => adjust end of existing mapping > + */> +++ b/filters/map/nbdkit-map-filter.pod > @@ -0,0 +1,173 @@ > +=head1 NAME > + > +nbdkit-map-filter - nbdkit map filter > + > +=head1 SYNOPSIS > + > + nbdkit --filter=map plugin map=FILENAME [plugin-args...] > + > +=head1 DESCRIPTION > + > +C<nbdkit-map-filter> is a filter that can serve an arbitrary map of > +regions of the underlying plugin. > + > +It is driven by a map file that contains a list of regions from the > +plugin and where they should be served in the output. > + > +For example this map would divide the plugin data into two 16K halves > +and swap them over: > + > + # map file > + 0,16K 16K # aaaaa > + 16K,16K 0 # bbbbb > + > +When visualised, this map file looks like: > + > + ┌──────────────┬──────────────┬─── ─ ─ ─ > + Plugin serves ... │ aaaaaaaaaaaa │ bbbbbbbbbbbb │ (extra data) > + │ 16K │ 16K │ > + └──────────────┴──────────────┴─── ─ ─ ─ > + │ │ > + Filter │ ┌─────────┘ > + transforms ... └──────────────┐ > + │ │ > + ┌──────────▼───┬─────▼────────┐ > + Client sees ... │ bbbbbbbbbbbb │ aaaaaaaaaaaa │ > + └──────────────┴──────────────┘ > + > +This is how to simulate L<nbdkit-offset-filter(1)> C<offset> and > +C<range> parameters: > + > + # offset,range > + 1M,32M 0 > + > + ┌─────┬─────────────────────┬─── ─ ─ ─ > + Plugin serves ... │ │ ccccccccccccccccccc │ (extra data) > + │ 1M │ 32M │ > + └─────┴─────────────────────┴─── ─ ─ ─ > + Filter │ > + transforms ... ┌─────┘ > + │ > + ┌─────────▼───────────┐ > + Client sees ... │ ccccccccccccccccccc │ > + └─────────────────────┘ > + > +You can also do obscure things like duplicating regions of the source: > + > + # map file > + 0,16K 0 > + 0,16K 16K > + > + ┌──────────────┬─── ─ ─ ─ > + Plugin serves ... │ aaaaaaaaaaaa │ (extra data) > + │ 16K │ > + └──────────────┴─── ─ ─ ─ > + Filter │ > + transforms ... └───┬──────────┐ > + │ │ > + ┌─────────▼────┬─────▼────────┐ > + Client sees ... │ aaaaaaaaaaaa │ aaaaaaaaaaaa │ > + └──────────────┴──────────────┘ > +When duplicating things, do we want to document that a single transaction is carried out in the order seen by the client (where aliases at later bytes overwrite any data written into the earlier alias in a long transaction), or do we want to put in hedge wording that (in the future) a request might be split into smaller regions that get operated on in parallel (thereby making the end contents indeterminate when writing to two aliases of the same byte in one transaction)?> +=head2 C<start-end> > + > + start-end offset > + > +means that the source region starting at byte C<start> through to byte > +C<end> (inclusive) is mapped to C<offset> through to > +C<offset+(end-start)> in the output. > + > +For example: > + > + 1024-2047 2048 > + > +maps the region starting at byte 1024 and ending at byte 2047 > +(inclusive) to bytes 2048-3071 in the output.Since you already support '2k', '2m' and such as shorthands for the start, is it worth creating a convenient shorthand for expressing '3M-1' for an end rather than having to write out 3145727?> + > +=head2 C<start> to end of plugin > + > + start- offset > + start offset > + > +If the C<end> field is omitted it means "up to the end of the > +underlying plugin". > + > +=head2 Size modifiers > + > +You can use the usual power-of-2 size modifiers like C<K>, C<M> etc. > + > +=head2 Overlapping mappings > + > +If there are multiple mappings in the map file that may apply to a > +particular byte of the filter output then it is the last one in the > +file which applies. > + > +=head2 Virtual size > + > +The virtual size of the filter output finishes at the last byte of the > +final mapped region. Note this is usually different from the size of > +the underlying plugin.Is there a syntax for explicitly adding an unmapped tail, to make the filter's output longer than the underlying plugin's size?> + > +=head2 Unmapped regions > + > +Any unmapped region (followed by a mapped region and therefore not > +beyond the virtual size) reads as zero and returns an error if > +written. > + > +Any mapping or part of a mapping where the source region refers beyond > +the end of the underlying plugin reads as zero and returns an error if > +written.Ah, so using the '(start,length) offset' entry does allow for an explicit unmapped tail at the end of the underlying plugin. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Reasonably Related Threads
- [PATCH v2 nbdkit 0/6] Add truncate filter and other fixes.
- [nbdkit PATCH 0/4] Fix truncate handling of real_size
- [PATCH nbdkit 3/4] Add map filter.
- [nbdkit PATCH 0/4] More mutex sanity checking
- [nbdkit PATCH] offset, partition: Fix .extents with non-zero offset