similar to: [nbdkit PATCH] extents: Cap maximum reply length

--- server/extents.c | 49 ++++++++++++++++++------------------------------ 1 file changed, 18 insertions(+), 31 deletions(-) diff --git a/server/extents.c b/server/extents.c index 2d609652..4ab5946c 100644 --- a/server/extents.c +++ b/server/extents.c @@ -42,6 +42,7 @@ #include <assert.h> #include "minmax.h" +#include "vector.h" #include "internal.h"

This tentative commit implements extents/can_extents, roughly as discussed in the previous thread here: https://www.redhat.com/archives/libguestfs/2019-March/msg00017.html I can't say that I'm a big fan of having the plugin allocate an extents array. There are no other plugin callbacks currently where we require the plugin to allocate complex data structures (or indeed do any allocation

On 3/28/19 11:18 AM, Richard W.M. Jones wrote: > This new pair of callbacks allows plugins to describe which extents in > the virtual disk are allocated, holes or zeroes. > --- > +++ b/server/extents.c > @@ -0,0 +1,210 @@ > +/* nbdkit > + * Copyright (C) 2019 Red Hat Inc. > + * All rights reserved. We started work on this phrase but it got waylaid by extents work; maybe

On 3/20/19 5:11 PM, Richard W.M. Jones wrote: > This pair of calls allows plugins to describe which extents in the > virtual disk are allocated, holes or zeroes. > --- I see you've already made a couple of tweaks to your block-status branch since posting this (skipping 0-legnth add_extent, and coalescing consecutive add_extents with the same type), but here's some more review:

We decided to not request the "base:allocation" context by default (if a client wants to use block_status on a different context, then they'd have to get any default request out of the way); however, block status is useless without at least one meta context. This adds a convenience knob for requesting that, and has the nice benefit of working with the --connect command line option

--- plugins/python/nbdkit-python-plugin.pod | 19 ++++++- plugins/python/python.c | 75 +++++++++++++++++++++++++ tests/test-python-plugin.py | 8 ++- tests/test_python.py | 73 +++++++++++++++++++++++- 4 files changed, 169 insertions(+), 6 deletions(-) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod

On Tue, Mar 12, 2019 at 11:20:25AM +0000, Richard W.M. Jones wrote: >This pair of calls allows plugins to describe which extents in the >virtual disk are allocated, holes or zeroes. >--- > docs/nbdkit-filter.pod | 22 +++++++++++ > docs/nbdkit-plugin.pod | 86 +++++++++++++++++++++++++++++++++++++++++ > include/nbdkit-common.h | 14 ++++++- > include/nbdkit-filter.h | 12 +++++-

Not sure what version we're up to, but this reimplements extents using the new simpler structure described in this thread: https://www.redhat.com/archives/libguestfs/2019-March/msg00077.html I also fixed most of the things that Eric pointed out in the previous review, although I need to go back over his replies and check I've got everything. This needs a bit more testing. However the

It is easy to demonstrate that our blocksize filter has a bug: if the minblock= setting is a higher granularity than the underlying plugin actually supports, and the client is trying to collect block status for the entire disk, a mid-block transition in the plugin can result in the filter rounding a request so small that it no longer makes progress, causing the client to see: nbd.Error:

Provide for modification of the limit of compressed extent size utilizing mount-time configuration settings. The size of compressed extents was limited to 128K, which leads to fragmentation of the extents (although the extents themselves may still be located contiguously). This limit is put in place to ease the RAM required when spreading compression across several CPUs, and to make sure the

I'm not sure exactly which version we're up to, but let's say it's version 4. I'm a lot happier with this version: - all filters have been reviewed and changed where I think that's necessary - can_extents is properly defined and implemented now - NBD protocol is followed - I believe it addresses all previous review points where possible The "only" thing

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

This was a rather longer trip around the houses than I anticipated! The basic purpose of the patch series is to set $nbdkit_stdio_safe to "0" or "1" in sh and eval plugin scripts. To do that, I ended up adding a nicer way to manipulate environ lists, and to do that, I ended up adding a whole generic vector implementation which is applicable in a lot of different places.

A compliant server should not send NBD_REPLY_TYPE_BLOCK_STATUS unless we successfully negotiated a meta context. And our default strictness settings refuse to let us send NBD_CMD_BLOCK_STATUS unless we negotiated a meta context. But when you mix non-default settings (using nbd_set_strict to disable STRICT_COMMANDS) to send a block status without having negotiated it, coupled with a non-compliant

This pair of calls allows plugins to describe which extents in the virtual disk are allocated, holes or zeroes. --- docs/nbdkit-filter.pod | 89 +++++++++++++++++++ docs/nbdkit-plugin.pod | 96 +++++++++++++++++++++ include/nbdkit-common.h | 10 ++- include/nbdkit-filter.h | 22 ++++- include/nbdkit-plugin.h | 6 +- server/internal.h | 4 + server/extents.c | 186

This new pair of callbacks allows plugins to describe which extents in the virtual disk are allocated, holes or zeroes. --- docs/nbdkit-filter.pod | 83 ++++++++++++++++ docs/nbdkit-plugin.pod | 97 +++++++++++++++++++ include/nbdkit-common.h | 10 +- include/nbdkit-filter.h | 22 ++++- include/nbdkit-plugin.h | 6 +- server/internal.h | 4 + server/extents.c | 210

Copies somewhat from the file plugin, with the difference that we always provide an .extents even if one or more of the split files does not support SEEK_HOLE. Testing is possible on a file system that supports sparse files when using nbdsh. Signed-off-by: Eric Blake <eblake@redhat.com> --- I'm going ahead and committing this, but am also posting this for review in case we later find

Last year we had a discussion[1] about how the hypercall ABI unfortunately contains fields that change width between 32- and 64-bit builds. This is a huge problem as we come up on the python management stack for ppc64, since the distributions ship 32-bit python. A 32-bit python/libxc cannot currently manage a 64-bit hypervisor. I had a patch but was unable to test it, and some other things were

On Sat, Mar 23, 2019 at 11:25:06AM -0500, Eric Blake wrote: > On 3/20/19 5:11 PM, Richard W.M. Jones wrote: > > + myfilter_extents (...) > > + { > > + size_t i; > > + struct nbdkit_extents *extents2; > > + struct nbdkit_extent e; > > + > > + extents2 = nbdkit_extents_new (offset); > > + next_ops->extents (nxdata, count, offset, flags,

This pair of calls allows plugins to describe which extents in the virtual disk are allocated, holes or zeroes. --- docs/nbdkit-filter.pod | 92 +++++++++++++++++++++ docs/nbdkit-plugin.pod | 99 +++++++++++++++++++++++ include/nbdkit-common.h | 10 ++- include/nbdkit-filter.h | 22 +++++- include/nbdkit-plugin.h | 6 +- server/internal.h | 4 + server/extents.c | 171