similar to: More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)

On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > Can we use something like the file plugin? thread pool of workers, > each keeping open vddk handle, and serving requests in parallel from > the same nbd socket? Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less

On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote: > On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > > Can we use something like the file plugin? thread pool of workers, > > > each keeping open vddk handle, and serving requests in parallel from >

On Wed, Aug 5, 2020 at 2:15 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > [NB: Adding PUBLIC mailing list because this is upstream discussion] > > On Mon, Aug 03, 2020 at 06:27:04PM +0100, Richard W.M. Jones wrote: > > On Mon, Aug 03, 2020 at 06:03:23PM +0300, Nir Soffer wrote: > > > On Mon, Aug 3, 2020 at 5:47 PM Richard W.M. Jones <rjones@redhat.com>

On Thu, Aug 6, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > See comment in code and > https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html > --- > plugins/vddk/vddk.c | 17 +++++++++++++---- > 1 file changed, 13 insertions(+), 4 deletions(-) > > diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c > index c24da96f..5926e181 100644

Nir, BTW what are you using for performance testing? As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v). Instead I've hacked up a copy of this program from libnbd: https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and-writes.c so

On Wed, Aug 5, 2020 at 4:28 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote: > > On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > > > Can we use something like the file plugin?

On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote: > > Can we use something like the file plugin? thread pool of workers, > > each keeping open vddk handle, and serving requests in parallel from > > the same nbd socket? > > Yes, but this isn't implemented in the

On Wed, Aug 5, 2020 at 3:47 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > Here are some results anyway. The command I'm using is: > > $ ./nbdkit -r -U - vddk \ > libdir=/path/to/vmware-vix-disklib-distrib \ > user=root password='***' \ > server='***' thumbprint=aa:bb:cc:... \ > vm=moref=3 \ >

I believe this roughly implements Nir's proposal here: https://www.redhat.com/archives/libguestfs/2020-August/msg00028.html Unfortunately it doesn't work for me. It actually slows things down quite a lot, for reasons I don't understand. Note the adjustment of the pool-max parameter and how it affects the total time. The results are quite reproducible. $ ./nbdkit -r -U - vddk

In theory this patch depends on this series: https://www.redhat.com/archives/libguestfs/2020-August/msg00021.html In practice I believe they're independent of each other, but the above series makes it easier to test. Rich.

Here are some results anyway. The command I'm using is: $ ./nbdkit -r -U - vddk \ libdir=/path/to/vmware-vix-disklib-distrib \ user=root password='***' \ server='***' thumbprint=aa:bb:cc:... \ vm=moref=3 \ file='[datastore1] Fedora 28/Fedora 28.vmdk' \ --run 'time /var/tmp/threaded-reads $unixsocket' Source for threaded-reads is

See comment in code and https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html --- plugins/vddk/vddk.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c index c24da96f..5926e181 100644 --- a/plugins/vddk/vddk.c +++ b/plugins/vddk/vddk.c @@ -478,11 +478,18 @@ vddk_dump_plugin (void) #endif } -/* XXX To

See comment in code and https://www.redhat.com/archives/libguestfs/2020-August/msg00023.html --- plugins/vddk/vddk.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c index 31e58f66..95e030d4 100644 --- a/plugins/vddk/vddk.c +++ b/plugins/vddk/vddk.c @@ -511,17 +511,29 @@ vddk_dump_plugin (void) #endif

On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote: > > I see, can change the python plugin to support multiple connections to imageio > > using SERIALIZE_REQUESTS? > > > > The GiL should not limit us since the GIL is released when you write to > > imageio socket, and

On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote: > I see, can change the python plugin to support multiple connections to imageio > using SERIALIZE_REQUESTS? > > The GiL should not limit us since the GIL is released when you write to > imageio socket, and this is likely where the plugin spends most of the time. It's an interesting question and one I'd not really

On Wed, Aug 5, 2020 at 3:28 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > Nir, BTW what are you using for performance testing? virt-v2v with local image, or imageio client with local image. > As far as I can tell it's not possible to make qemu-img convert use > multi-conn when connecting to the source (which is going to be a > problem if we want to use this

The pool is only used for readonly connections, since writable connections usually take a lock on the server side and therefore you cannot open more than one. --- plugins/vddk/nbdkit-vddk-plugin.pod | 7 + plugins/vddk/vddk.c | 201 ++++++++++++++++++++++------ 2 files changed, 164 insertions(+), 44 deletions(-) diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod

On Thu, Aug 6, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > The pool is only used for readonly connections, since writable > connections usually take a lock on the server side and therefore you > cannot open more than one. > --- > plugins/vddk/nbdkit-vddk-plugin.pod | 7 + > plugins/vddk/vddk.c | 201 ++++++++++++++++++++++------ > 2

We had a query yesterday about the VDDK plugin and making it actually obey the weird "Multithreading Considerations" rules in the VDDK documentation (https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf) This patch is my attempt to implement this. The idea is that the

On Fri, Aug 02, 2019 at 02:26:16PM -0500, Eric Blake wrote: > When we first created the sh plugin, we copied the thread model used > in most other language bindings of SERIALIZE_ALL_REQUESTS, because it > was easier to reason about and there was no way for a script to > override our choice. However, we've since added support for scripts > to request a tighter model when needed,