similar to: [nbdkit PATCH v2] Introduce cacheextents filter

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_on=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map` runs

On Thu, May 16, 2019 at 08:05:18AM -0500, Eric Blake wrote: >On 5/15/19 5:39 PM, Martin Kletzander wrote: >> This filter caches the last result of the extents() call and offers a nice >> speed-up for clients that only support req_on=1 in combination with plugins like > >s/on/one/ > >> vddk, which has no overhead for returning information for multiple extents in

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_one=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map`

On 5/15/19 5:39 PM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_on=1 in combination with plugins like s/on/one/ > vddk, which has no overhead for returning information for multiple extents in > one call, but that call is very time-consuming. > > Quick test showed that on a

On 6/11/19 3:49 AM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_one=1 in combination with plugins > like vddk, which has no overhead for returning information for multiple extents > in one call, but that call is very time-consuming. > > Quick test showed that on a fast

On Wed, May 15, 2019 at 02:54:17PM +0200, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_on=1 in combination with plugins like > vddk, which has no overhead for returning information for multiple extents in > one call, but that call is very time-consuming. > > Quick test showed

On 5/20/19 8:23 AM, Martin Kletzander wrote: >>> +  if (cache_extents && >>> +      offset >= cache_start && offset < cache_end) { >>> +    nbdkit_debug ("cacheextents: returning from cache"); >>> +    return cacheextents_add (extents, err); >>> +  } >>> + >>> +  nbdkit_debug ("cacheextents: cache

Apply the attached patch on top of yours which contains miscellaneous fixes. If you use ‘./configure --enable-gcc-warnings’ then it will enable GCC warnings. The test doesn't really test anything. If extents are being cached then I have two ideas about how you could see that: Either (1) you could look at debug messages, the second extents call shouldn't be processed at the plugin layer.

On Thu, May 16, 2019 at 08:42:43AM +0100, Richard W.M. Jones wrote: > >Apply the attached patch on top of yours which contains miscellaneous >fixes. If you use ‘./configure --enable-gcc-warnings’ then it will >enable GCC warnings. > Thanks. Again, too much used to libvirt codebase where those are being enabled by default for builds from git. >The test doesn't really test

On 6/11/19 3:49 AM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_one=1 in combination with plugins > like vddk, which has no overhead for returning information for multiple extents > in one call, but that call is very time-consuming. > > Quick test showed that on a fast

Ultimately, both the blocksize and swab filters want to return aligned extents to the client. I'm posting this as a snapshot of my work in progress on how I plan to get there (it's not quite working yet, but I'm done for today and wanted to at least document my ideas). I might also add a convenience function for nbdkit_extents_offset, since we have a number of filters that repeat the

When querying the extents of the underlying plugin, we should only translate the starting offset, and let the plugin report for at least as many bytes as our range permits. Otherwise, short-changing the range causes bad behavior such as returning 0 extents, or even failing the creation of an extents tracker: $ cat script case "$1" in get_size) echo 1m;; can_extents) ;;

We documented our intent of only allowing a filter to run with the same version of nbdkit it was compiled against, but up to now, were not actually enforcing that - we had only been insisting on the looser notion of a matching ._api_version, which doesn't help when we've forgotten to bump that macro when making incompatible API/ABI changes (see commit 6934d4c1). However, we can't use

This tentative commit implements extents/can_extents, roughly as discussed in the previous thread here: https://www.redhat.com/archives/libguestfs/2019-March/msg00017.html I can't say that I'm a big fan of having the plugin allocate an extents array. There are no other plugin callbacks currently where we require the plugin to allocate complex data structures (or indeed do any allocation

Includes a rework of the previously posted patch for --run improvements (mostly with improved comments and commit message; I decided that waiting for the captive nbdkit to exit was overkill), and four new patches. The tests are intentionally separate, to allow rearranging the order of the series to see the failures being fixed. Eric Blake (6): server: Propagate unexpected nbdkit failure with

It does not need configuring and speeds up the process of requesting extents if the client asks for them one by one (like qemu-img). Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- Essentially a v2 of the following patch since now there is a probing function: https://www.redhat.com/archives/libguestfs/2019-September/msg00249.html v2v/nbdkit.ml | 7 +++++++ 1 file changed, 7

On 4/23/19 2:06 PM, Eric Blake wrote: > Now that cleanup.h is in common code, we can use it in our > filters. The first round focuses just on places that called > nbdkit_extents_free(), as all three callers had multiple exit paths > that definitely benefit from the macro. > > Signed-off-by: Eric Blake <eblake@redhat.com> > --- > filters/offset/offset.c | 13

On Tue, Jul 07, 2020 at 05:22:46PM -0500, Eric Blake wrote: [...] > +/* Compute aligned extents on behalf of a filter. */ > +int > +nbdkit_extents_aligned (struct nbdkit_next_ops *next_ops, > + nbdkit_backend *nxdata, > + uint32_t count, uint64_t offset, > + uint32_t flags, uint32_t align, > +

I'm not sure exactly which version we're up to, but let's say it's version 4. I'm a lot happier with this version: - all filters have been reviewed and changed where I think that's necessary - can_extents is properly defined and implemented now - NBD protocol is followed - I believe it addresses all previous review points where possible The "only" thing

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.