similar to: [nbdkit PATCH] Introduce cacheextents filter

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_one=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map`

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_on=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map` runs

On Thu, May 16, 2019 at 08:05:18AM -0500, Eric Blake wrote: >On 5/15/19 5:39 PM, Martin Kletzander wrote: >> This filter caches the last result of the extents() call and offers a nice >> speed-up for clients that only support req_on=1 in combination with plugins like > >s/on/one/ > >> vddk, which has no overhead for returning information for multiple extents in

On 6/11/19 3:49 AM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_one=1 in combination with plugins > like vddk, which has no overhead for returning information for multiple extents > in one call, but that call is very time-consuming. > > Quick test showed that on a fast

On Wed, May 15, 2019 at 02:54:17PM +0200, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_on=1 in combination with plugins like > vddk, which has no overhead for returning information for multiple extents in > one call, but that call is very time-consuming. > > Quick test showed

On 5/15/19 5:39 PM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_on=1 in combination with plugins like s/on/one/ > vddk, which has no overhead for returning information for multiple extents in > one call, but that call is very time-consuming. > > Quick test showed that on a

On 5/20/19 8:23 AM, Martin Kletzander wrote: >>> +  if (cache_extents && >>> +      offset >= cache_start && offset < cache_end) { >>> +    nbdkit_debug ("cacheextents: returning from cache"); >>> +    return cacheextents_add (extents, err); >>> +  } >>> + >>> +  nbdkit_debug ("cacheextents: cache

On 6/11/19 3:49 AM, Martin Kletzander wrote: > This filter caches the last result of the extents() call and offers a nice > speed-up for clients that only support req_one=1 in combination with plugins > like vddk, which has no overhead for returning information for multiple extents > in one call, but that call is very time-consuming. > > Quick test showed that on a fast

Apply the attached patch on top of yours which contains miscellaneous fixes. If you use ‘./configure --enable-gcc-warnings’ then it will enable GCC warnings. The test doesn't really test anything. If extents are being cached then I have two ideas about how you could see that: Either (1) you could look at debug messages, the second extents call shouldn't be processed at the plugin layer.

On Thu, May 16, 2019 at 08:42:43AM +0100, Richard W.M. Jones wrote: > >Apply the attached patch on top of yours which contains miscellaneous >fixes. If you use ‘./configure --enable-gcc-warnings’ then it will >enable GCC warnings. > Thanks. Again, too much used to libvirt codebase where those are being enabled by default for builds from git. >The test doesn't really test

Not sure what version we're up to, but this reimplements extents using the new simpler structure described in this thread: https://www.redhat.com/archives/libguestfs/2019-March/msg00077.html I also fixed most of the things that Eric pointed out in the previous review, although I need to go back over his replies and check I've got everything. This needs a bit more testing. However the

This fix isn't exhaustive but it fixes some obvious problems in the filters. Rich.

I'm not sure exactly which version we're up to, but let's say it's version 4. I'm a lot happier with this version: - all filters have been reviewed and changed where I think that's necessary - can_extents is properly defined and implemented now - NBD protocol is followed - I believe it addresses all previous review points where possible The "only" thing

I pushed a few of the msys2 patches upstream. I changed the way that $(SHARED_LDFLAGS) works so it's more to my liking, and the others were pushed unchanged. Three patches remain which I'm posting on the mailing list for proper review. Rich.

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

It does not need configuring and speeds up the process of requesting extents if the client asks for them one by one (like qemu-img). Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- v2v/input_libvirt_vddk.ml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/v2v/input_libvirt_vddk.ml b/v2v/input_libvirt_vddk.ml index a8e4fd5ac78e..cc63f77ce825 100644 ---

It does not need configuring and speeds up the process of requesting extents if the client asks for them one by one (like qemu-img). Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- Essentially a v2 of the following patch since now there is a probing function: https://www.redhat.com/archives/libguestfs/2019-September/msg00249.html v2v/nbdkit.ml | 7 +++++++ 1 file changed, 7

When querying the extents of the underlying plugin, we should only translate the starting offset, and let the plugin report for at least as many bytes as our range permits. Otherwise, short-changing the range causes bad behavior such as returning 0 extents, or even failing the creation of an extents tracker: $ cat script case "$1" in get_size) echo 1m;; can_extents) ;;

On 4/23/19 2:06 PM, Eric Blake wrote: > Now that cleanup.h is in common code, we can use it in our > filters. The first round focuses just on places that called > nbdkit_extents_free(), as all three callers had multiple exit paths > that definitely benefit from the macro. > > Signed-off-by: Eric Blake <eblake@redhat.com> > --- > filters/offset/offset.c | 13

I pushed this one already. It should go into the stable-1.12 branch too. Rich.