similar to: [PATCH nbdkit] vddk: Relax threading model and enable multi-conn.

I believe this roughly implements Nir's proposal here: https://www.redhat.com/archives/libguestfs/2020-August/msg00028.html Unfortunately it doesn't work for me. It actually slows things down quite a lot, for reasons I don't understand. Note the adjustment of the pool-max parameter and how it affects the total time. The results are quite reproducible. $ ./nbdkit -r -U - vddk

On Thu, Aug 6, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > See comment in code and > https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html > --- > plugins/vddk/vddk.c | 17 +++++++++++++---- > 1 file changed, 13 insertions(+), 4 deletions(-) > > diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c > index c24da96f..5926e181 100644

See comment in code and https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html --- plugins/vddk/vddk.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c index 31e58f66..95e030d4 100644 --- a/plugins/vddk/vddk.c +++ b/plugins/vddk/vddk.c @@ -511,17 +511,29 @@ vddk_dump_plugin (void) #endif

We had a query yesterday about the VDDK plugin and making it actually obey the weird "Multithreading Considerations" rules in the VDDK documentation (https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf) This patch is my attempt to implement this. The idea is that the

Couple of fixes to return correct data and one nice-to-have clean-up which is not needed. I just find it nicer to read. Martin Kletzander (3): vddk: Use a separate handle for single-link=true vddk: Do not report hole extents to be zero with single-link=true vddk: Eliminate one needless goto plugins/vddk/vddk.c | 48 +++++++++++++++++++++++++++++++++------------ 1 file changed, 36

On Thu, Aug 6, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > The pool is only used for readonly connections, since writable > connections usually take a lock on the server side and therefore you > cannot open more than one. > --- > plugins/vddk/nbdkit-vddk-plugin.pod | 7 + > plugins/vddk/vddk.c | 201 ++++++++++++++++++++++------ > 2

This is my second attempt at this. The first version (also not working) was here: https://www.redhat.com/archives/libguestfs/2019-October/msg00062.html In part 1/2 I introduce a new .ready_to_serve plugin method which is called after forking and just before accepting any client connection. The idea would be that plugins could start background threads here. However this doesn't work well in

The pool is only used for readonly connections, since writable connections usually take a lock on the server side and therefore you cannot open more than one. --- plugins/vddk/nbdkit-vddk-plugin.pod | 7 + plugins/vddk/vddk.c | 201 ++++++++++++++++++++++------ 2 files changed, 164 insertions(+), 44 deletions(-) diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod

See comment in code and https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html --- plugins/vddk/vddk.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c index c24da96f..5926e181 100644 --- a/plugins/vddk/vddk.c +++ b/plugins/vddk/vddk.c @@ -478,11 +478,18 @@ vddk_dump_plugin (void) #endif } -/* XXX To

When using VIXDISKLIB_FLAG_OPEN_SINGLE_LINK, parent images are not taken into account. However, the data we get from VixDiskLib_QueryAllocatedBlocks() has such granularity that it might return incorrect information. For example when the top image has only one block allocated, we get information about the whole chunk being allocated. This results in the consequent read to return invalid data -

On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > Can we use something like the file plugin? thread pool of workers, > each keeping open vddk handle, and serving requests in parallel from > the same nbd socket? Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less

Since it is required, it might as well be magic ;) Signed-off-by: Eric Blake <eblake@redhat.com> --- As written, applies on top of my v3 patch for dlopen; but it would be easy enough to rebase and take this one now even if we aren't sure about the dlopen stuff plugins/vddk/nbdkit-vddk-plugin.pod | 5 ++++- plugins/vddk/vddk.c | 3 ++- tests/test-vddk.sh

On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote: > On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > > Can we use something like the file plugin? thread pool of workers, > > > each keeping open vddk handle, and serving requests in parallel from >

On 8/5/20 7:48 AM, Richard W.M. Jones wrote: > See comment in code and > https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html > --- > plugins/vddk/vddk.c | 32 ++++++++++++++++++++++++++++---- > 1 file changed, 28 insertions(+), 4 deletions(-) I'll have to trust your testing, but code-wise, this looks safe. -- Eric Blake, Principal Software Engineer Red

On Thu, Aug 6, 2020 at 7:24 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Thu, Aug 06, 2020 at 06:46:06PM +0300, Nir Soffer wrote: > > Given the poor results, I suspect that that handles created using same > > connection share a lock. This also makes sense if connection abstract a > > blocking socket. > > Nice theory, but it didn't work. Again

I like this change. I think we were overloading the config_complete method before to do two different things (complete configuration; do any allocation/housekeeping necessary before we can start serving). The only questions in my mind are whether we want this before 1.18, and whether the name ("get_ready") is a good one. Rich.

On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > Can we use something like the file plugin? thread pool of workers, > > each keeping open vddk handle, and serving requests in parallel from > > the same nbd socket? > > Yes, but this isn't implemented in the

On Wed, Aug 05, 2020 at 08:12:26AM -0500, Eric Blake wrote: > On 8/5/20 7:48 AM, Richard W.M. Jones wrote: > >See comment in code and > >https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html > >--- > > plugins/vddk/vddk.c | 32 ++++++++++++++++++++++++++++---- > > 1 file changed, 28 insertions(+), 4 deletions(-) > > I'll have to trust

Probably needs a bit of cleanup, but seems like it is generally the right direction. One thing I've noticed is that the expect test randomly (but rarely) hangs :-( I guess something is racey but I don't know what at the moment. Rich.

On Wed, Aug 5, 2020 at 4:28 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote: > > On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > > > Can we use something like the file plugin?