search for: real_siz

The truncate filter tried to be careful to lock access to setting or reading the real_size variable learned from calling next_ops->get_size, in anticipation of not behaving incorrectly if the NBD protocol makes dynamic resize supported, and where the global variable could serve as a cache rather than having to always call next_ops->get_size to recompute things. However, nbdkit-plu...

Add a helper function get_real_size() to make it easier to add sanity checking that mutex calls don't fail. Signed-off-by: Eric Blake <eblake@redhat.com> --- I'm a bit unsure why truncate.c needs a lock anyway. I guess it's because we are storing 'size' and 'real_size' as globals rather than per-...

While working on adding assertions to pthread_mutex_lock calls, I noticed that the truncate filter's use of mutex didn't really protect us, and isn't really necessary. Cleaning that up also spotted a couple of other potential cleanups. Eric Blake (4): filters: Drop useless .open callbacks truncate: Fix corruption when plugin changes per-connection size truncate: Test for safe

On Wed, Apr 24, 2019 at 05:24:39PM -0500, Eric Blake wrote: > Add a helper function get_real_size() to make it easier to add sanity > checking that mutex calls don't fail. This patch is fine, ACK. > I'm a bit unsure why truncate.c needs a lock anyway. I guess it's > because we are storing 'size' and 'real_size' as globals rather than > per-connection v...

...; The start offset to serve (default 0).\n" \ + "range=<LENGTH> The total size to serve (default rest of file)." /* Get the file size. */ static int64_t @@ -82,16 +75,23 @@ offset_get_size (struct nbdkit_next_ops *next_ops, void *nxdata, { int64_t real_size = next_ops->get_size (nxdata); + if (real_size == -1) + return -1; + if (range >= 0) { - if (offset + range > real_size) { + if (offset > real_size - range) { nbdkit_error ("offset+range is larger than the real size " "of the und...

...offset.c index efe5c6d1..00122770 100644 --- a/filters/offset/offset.c +++ b/filters/offset/offset.c @@ -140,9 +140,9 @@ offset_extents (struct nbdkit_next_ops *next_ops, void *nxdata, size_t i; CLEANUP_EXTENTS_FREE struct nbdkit_extents *extents2 = NULL; struct nbdkit_extent e; - int64_t real_size = next_ops->get_size (nxdata); + int64_t real_size = range >= 0 ? offset + range : next_ops->get_size (nxdata); - extents2 = nbdkit_extents_new (offs + offset, real_size - offset); + extents2 = nbdkit_extents_new (offs + offset, real_size); if (extents2 == NULL) { *err = errno...

...lude <string.h> +#include <errno.h> + +#include <nbdkit-filter.h> + +#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL + +/* These are the parameters. */ +static int64_t truncate = -1, round_up = -1, round_down = -1; + +/* The real size of the underlying plugin. */ +static int64_t real_size; + +/* The calculated size after applying the parameters. */ +static int64_t size; + +/* Called for each key=value passed on the command line. */ +static int +truncate_config (nbdkit_next_config *next, void *nxdata, + const char *key, const char *value) +{ + if (strcmp (key, &quot...

I do have a question about whether patch 2 is right, or whether I've exposed a bigger problem in the truncate (and possibly other) filter, but the rest seem fairly straightforward. Eric Blake (4): server: Check for pthread lock failures truncate: Factor out reading real_size under mutex plugins: Check for mutex failures filters: Check for mutex failures filters/cache/cache.c | 23 +++++++++---------- filters/cow/cow.c | 19 +++++++--------- filters/error/error.c | 7 +++--- filters/log/log.c | 3 +-- filters/rate/rate.c...

...+ +#include "ispowerof2.h" +#include "iszero.h" + +#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL + +/* These are the parameters. */ +static int64_t truncate = -1; +static unsigned round_up = 0, round_down = 0; + +/* The real size of the underlying plugin. */ +static uint64_t real_size; + +/* The calculated size after applying the parameters. */ +static uint64_t size; + +/* This lock protects the real_size and size fields. */ +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; + +static int +parse_round_param (const char *key, const char *value, unsigned *ret) +{ + int64_...

...ust be a power of 2. > + > +#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL > + > +/* These are the parameters. */ > +static int64_t truncate = -1; > +static unsigned round_up = 0, round_down = 0; > + > +/* The real size of the underlying plugin. */ > +static uint64_t real_size; > + > +/* The calculated size after applying the parameters. */ > +static uint64_t size; > + > +/* This lock protects the real_size and size fields. */ > +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; Do we need the lock, or can we rely on the fact that .prepare will...

...10 +81,10 @@ offset_config_complete (nbdkit_next_config_complete *next, void *nxdata) /* Get the file size. */ static int64_t -offset_get_size (struct nbdkit_next *next, void *nxdata, +offset_get_size (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle) { - int64_t real_size = next->get_size (nxdata); + int64_t real_size = next_ops->get_size (nxdata); if (range >= 0) { if (offset + range > real_size) { @@ -99,35 +99,35 @@ offset_get_size (struct nbdkit_next *next, void *nxdata, /* Read data. */ static int -offset_pread (struct nbdkit_next *n...

...it_next_ops *next_ops, void *nxdata, + void *handle, uint32_t count, uint64_t offs, uint32_t flags, + struct nbdkit_extents *extents, int *err) +{ + struct handle *h = handle; + size_t i; + struct nbdkit_extents *extents2; + struct nbdkit_extent e; + int64_t real_size = next_ops->get_size (nxdata); + + extents2 = nbdkit_extents_new (offs + h->offset, real_size - h->offset); + if (extents2 == NULL) { + *err = errno; + return -1; + } + if (next_ops->extents (nxdata, count, offs + h->offset, + flags, extents2, err)...

...tic int +offset_extents (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, uint32_t count, uint64_t offs, uint32_t flags, + struct nbdkit_extents *extents, int *err) +{ + size_t i; + struct nbdkit_extents *extents2; + struct nbdkit_extent e; + int64_t real_size = next_ops->get_size (nxdata); + + extents2 = nbdkit_extents_new (offs + offset, real_size - offset); + if (extents2 == NULL) { + *err = errno; + return -1; + } + if (next_ops->extents (nxdata, count, offs + offset, + flags, extents2, err) == -1) + goto e...

....M. Jones wrote: > Allows you to safely use nbdkit-offset-filter on top of a plugin > supporting extents. > --- > filters/offset/offset.c | 36 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 36 insertions(+) > > + > + extents2 = nbdkit_extents_new (offs + offset, real_size - offset); > + if (extents2 == NULL) { > + *err = errno; > + return -1; > + } Here, we are careful to set *err. > + if (next_ops->extents (nxdata, count, offs + offset, > + flags, extents2, err) == -1) > + goto error; And here. > +...

...ata, >+ void *handle, uint32_t count, uint64_t offs, uint32_t flags, >+ struct nbdkit_extents *extents, int *err) >+{ >+ struct handle *h = handle; >+ size_t i; >+ struct nbdkit_extents *extents2; >+ struct nbdkit_extent e; >+ int64_t real_size = next_ops->get_size (nxdata); >+ >+ extents2 = nbdkit_extents_new (offs + h->offset, real_size - h->offset); >+ if (extents2 == NULL) { >+ *err = errno; >+ return -1; >+ } >+ if (next_ops->extents (nxdata, count, offs + h->offset, >+...

This patch series proposes two new filters. * truncate: This can truncate, extend, round up or round down the size of a plugin/device. A typical usage is to fix the qemu problem that it can only handle devices which are a multiple of 512-bytes: nbdkit --filter=truncate random size=500 round-up=512 This will serve a virtual device with size 512 bytes. Reading from the last 12 bytes will

...1; @@ -138,7 +140,7 @@ offset_extents (struct nbdkit_next_ops *next_ops, void *nxdata, struct nbdkit_extents *extents, int *err) { size_t i; - struct nbdkit_extents *extents2; + CLEANUP_EXTENTS_FREE struct nbdkit_extents *extents2 = NULL; struct nbdkit_extent e; int64_t real_size = next_ops->get_size (nxdata); @@ -149,20 +151,15 @@ offset_extents (struct nbdkit_next_ops *next_ops, void *nxdata, } if (next_ops->extents (nxdata, count, offs + offset, flags, extents2, err) == -1) - goto error; + return -1; for (i = 0; i < nb...

...;offset=<OFFSET> (required) The start offset to serve.\n" \ + "range=<LENGTH> The total size to serve." + +/* Get the file size. */ +static int64_t +offset_get_size (struct nbdkit_next *next, void *nxdata, + void *handle) +{ + int64_t real_size = next->get_size (nxdata); + + if (range >= 0) { + if (offset + range > real_size) { + nbdkit_error ("offset + range is larger than the real size of the underlying file or device"); + return -1; + } + return range; + } + else + return real_size - offset;...

I have dropped the map filter from this series for now while I try to get it working. However I think the truncate filter is in a good shape. This incorporates all feedback from Eric's review. Also there are three small fixes to the filter code, all revealed when I was testing using multiple filters which we'd not done much of before. Rich.

...void *nxdata, return 0; } +/* Extents. */ +static int +truncate_extents (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, uint32_t count, uint64_t offset, + uint32_t flags, struct nbdkit_extents *extents, int *err) +{ + uint32_t n; + uint64_t real_size_copy; + struct nbdkit_extents *extents2; + size_t i; + + pthread_mutex_lock (&lock); + real_size_copy = real_size; + pthread_mutex_unlock (&lock); + + /* If the entire request is beyond the end of the underlying plugin + * then this is the easy case: return a hole. + */ + if (of...