Libguestfs - Aug 2020 - Re: More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)

On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer
wrote:
> I see, can change the python plugin to support multiple connections to
imageio
> using SERIALIZE_REQUESTS?
> 
> The GiL should not limit us since the GIL is released when you write to
> imageio socket, and this is likely where the plugin spends most of the
time.
It's an interesting question and one I'd not really considered at all.
Does the Python GIL actively mutex different threads if they call into
Python code at the same time?  If it's truly a lock, then it should,
in which case it should be safe to change the Python plugin to
PARALLEL ...

I'll try it out and get back to you.
> I'm not sure what triggers using multiple connections in qemu-img and
> how it decide how many connections should be used, but we report
> the number of writers in OPTIONS:
> http://ovirt.github.io/ovirt-imageio/images.html#max_writers
> 
> There is a hard limit in vdsm, because it runs qemu-nbd with
> --shared=8, so you should
> not use more than 8 connections, they will just block on qemu-nbd forever.
It's different for qemu NBD client and server.  Eric told me on IRC a
few minutes ago that qemu NBD client does not "yet" support
multi-conn.  However it is implemented by qemu-nbd (the server) for
readonly connections.

Note that multi-conn and --shared are (a bit) different.  Multi-conn
is a flag which the server can send to clients to indicate not only
that multiple connections are possible, but also that they are come
with certain guarantees:

  bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
  entirely without cache, or that the cache it uses is shared among
  all connections to the given device. In particular, if this flag is
  present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
  be visible across all connections when the server sends its reply to
  that command to the client. In the absence of this flag, clients
  SHOULD NOT multiplex their commands over more than one connection to
  the export.

“--shared” limits the number of connections that the server will
accept at the same time (like the nbdkit limit filter).  But it would
still be possible for an NBD server to accept multiple connections
from a single client but not be multi-conn safe.

Also NBD lets you multiplex commands on a single connection (which
does not require multi-conn or --shared).

BTW I found that multi-conn is a big win with the Linux kernel NBD
client.
> We use 4 connections by default, giving about 100% speed up compared
> with one connection. 2 connections give about 80% speed up.  If the
> number of connections is related to the number of coroutines, you
> can use -m 4 to use 4 coroutines.
>
> Using -W will improve performance. In this mode every coroutine will
> do the I/O when it is ready, instead of waiting for other coroutines
> and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do
for qemu NBD client.  Maybe improve multiplexing?  They don't enable
multi-conn :-(

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
> > I see, can change the python plugin to support multiple connections to
imageio
> > using SERIALIZE_REQUESTS?
> >
> > The GiL should not limit us since the GIL is released when you write
to
> > imageio socket, and this is likely where the plugin spends most of the
time.
>
> It's an interesting question and one I'd not really considered at
all.
> Does the Python GIL actively mutex different threads if they call into
> Python code at the same time?
Yes, only one thread can run python code at the same time, there is no
way around this. But before calling most syscalls, python releases the GIL.
When the syscall returns, python acquires the GIL again.

But thread can be switched at any point in time, so must use locking
in the python if you modify or access shared state.
>  If it's truly a lock, then it should,
> in which case it should be safe to change the Python plugin to
> PARALLEL ...
Using multiple threads on the same http socket will not work, you must have
multiple sockets to imageio.
>
> I'll try it out and get back to you.
>
> > I'm not sure what triggers using multiple connections in qemu-img
and
> > how it decide how many connections should be used, but we report
> > the number of writers in OPTIONS:
> > http://ovirt.github.io/ovirt-imageio/images.html#max_writers
> >
> > There is a hard limit in vdsm, because it runs qemu-nbd with
> > --shared=8, so you should
> > not use more than 8 connections, they will just block on qemu-nbd
forever.
>
> It's different for qemu NBD client and server.  Eric told me on IRC a
> few minutes ago that qemu NBD client does not "yet" support
> multi-conn.  However it is implemented by qemu-nbd (the server) for
> readonly connections.
>
> Note that multi-conn and --shared are (a bit) different.  Multi-conn
> is a flag which the server can send to clients to indicate not only
> that multiple connections are possible, but also that they are come
> with certain guarantees:
>
>   bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
>   entirely without cache, or that the cache it uses is shared among
>   all connections to the given device. In particular, if this flag is
>   present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
>   be visible across all connections when the server sends its reply to
>   that command to the client. In the absence of this flag, clients
>   SHOULD NOT multiplex their commands over more than one connection to
>   the export.
>
> “--shared” limits the number of connections that the server will
> accept at the same time (like the nbdkit limit filter).  But it would
> still be possible for an NBD server to accept multiple connections
> from a single client but not be multi-conn safe.
>
> Also NBD lets you multiplex commands on a single connection (which
> does not require multi-conn or --shared).
>
> BTW I found that multi-conn is a big win with the Linux kernel NBD
> client.
>
> > We use 4 connections by default, giving about 100% speed up compared
> > with one connection. 2 connections give about 80% speed up.  If the
> > number of connections is related to the number of coroutines, you
> > can use -m 4 to use 4 coroutines.
> >
> > Using -W will improve performance. In this mode every coroutine will
> > do the I/O when it is ready, instead of waiting for other coroutines
> > and submit the I/O in the right order.
>
> I think Eric might have a better idea about what -m and -W really do
> for qemu NBD client.  Maybe improve multiplexing?  They don't enable
> multi-conn :-(
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>

On 8/5/20 9:10 AM, Richard W.M. Jones wrote:
> On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
>> I see, can change the python plugin to support multiple connections to
imageio
>> using SERIALIZE_REQUESTS?
>>
>> The GiL should not limit us since the GIL is released when you write to
>> imageio socket, and this is likely where the plugin spends most of the
time.
> 
> It's an interesting question and one I'd not really considered at
all.
> Does the Python GIL actively mutex different threads if they call into
> Python code at the same time?  If it's truly a lock, then it should,
> in which case it should be safe to change the Python plugin to
> PARALLEL ...
> 
> I'll try it out and get back to you.
Yeah, I would not be surprised if we could make the Python plugin more 
performant, but matching our glue code to the python docs for embedding 
with C code is  not trivial, so I haven't spent the time trying.

> Also NBD lets you multiplex commands on a single connection (which
> does not require multi-conn or --shared).
> 
> BTW I found that multi-conn is a big win with the Linux kernel NBD
> client.
> 
>> We use 4 connections by default, giving about 100% speed up compared
>> with one connection. 2 connections give about 80% speed up.  If the
>> number of connections is related to the number of coroutines, you
>> can use -m 4 to use 4 coroutines.
>>
>> Using -W will improve performance. In this mode every coroutine will
>> do the I/O when it is ready, instead of waiting for other coroutines
>> and submit the I/O in the right order.
> 
> I think Eric might have a better idea about what -m and -W really do
> for qemu NBD client.  Maybe improve multiplexing?  They don't enable
> multi-conn :-(
Correct.  Using -W doesn't make sense without -m (if you only have one 
worker, you might as well proceed linearly than trying to randomize 
access, but even when you have multiple threads, there are cases where 
linear operations are still useful, such as 'nbdkit streaming'.  But -m 
is definitely the knob that controls how many outstanding I/O requests 
qemu-img is willing to use; and once you are using -m, using -W makes 
life easier for those coroutines to stay active.  The default -m1 says 
that at most one request is outstanding, so parallelism in the server is 
not utilized.  With higher -m, qemu-img issues up to that many requests 
without waiting for server answers, but all on the same NBD connection. 
Ideally, you'll get the maximum behavior as 'qemu-img -m' and
'nbdkit
--threads' choose the same values; if either side has fewer in-flight 
operations permitted than the other, then that side has the potential to 
become a bottleneck.  Right now, nbdkit defaults to 16 threads (that is, 
up to 16 in-flight operations) for any PARALLEL plugin.

And someday, I'd love to improve nbdkit's PARALLEL mode to make its 
thread-pool more of an on-demand setup (right now, we pre-create all 16 
threads up front, even if the client never reaches 16 in-flight 
operations at once, which is a bit wasteful), but other than potential 
performance improvements, it should be a transparent change to both 
plugins and clients.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On 8/5/20 9:38 AM, Eric Blake wrote:
> Correct.  Using -W doesn't make sense without -m (if you only have one 
> worker, you might as well proceed linearly than trying to randomize 
> access, but even when you have multiple threads, there are cases where 
> linear operations are still useful, such as 'nbdkit streaming'. 
But -m
> is definitely the knob that controls how many outstanding I/O requests 
> qemu-img is willing to use; and once you are using -m, using -W makes 
> life easier for those coroutines to stay active.  The default -m1 says 
> that at most one request is outstanding, so parallelism in the server is 
> not utilized.  With higher -m, qemu-img issues up to that many requests 
> without waiting for server answers, but all on the same NBD connection. 
> Ideally, you'll get the maximum behavior as 'qemu-img -m' and
'nbdkit
> --threads' choose the same values; if either side has fewer in-flight 
> operations permitted than the other, then that side has the potential to 
> become a bottleneck.  Right now, nbdkit defaults to 16 threads (that is, 
> up to 16 in-flight operations) for any PARALLEL plugin.
This reminds me of a previous thread: we probably want to add a new mode 
to nbdkit parallelism:
https://www.redhat.com/archives/libguestfs/2019-May/msg00089.html

Use of qemu-nbd -m without -W is like my proposed 
NBDKIT_THREAD_MODEL_SERIALIZE_RETIREMENT (okay, I just invented that 
name; I did not name it in that thread).  Use of qemu-nbd -m with -W is 
then NBDKIT_THREAD_MODEL_PARALLEL.  Use of qemu-nbd without -m (at which 
point -W makes no difference) is like 
NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS.  The middle mode is where you 
allow multiple in-flight requests, but serialize their retirement 
(different than serializing their issuance), for a client that can't 
deal with out-of-order replies.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On Wed, Aug 5, 2020 at 5:38 PM Eric Blake <eblake@redhat.com>
wrote:
>
> On 8/5/20 9:10 AM, Richard W.M. Jones wrote:
> > On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
> >> I see, can change the python plugin to support multiple
connections to imageio
> >> using SERIALIZE_REQUESTS?
> >>
> >> The GiL should not limit us since the GIL is released when you
write to
> >> imageio socket, and this is likely where the plugin spends most of
the time.
> >
> > It's an interesting question and one I'd not really considered
at all.
> > Does the Python GIL actively mutex different threads if they call into
> > Python code at the same time?  If it's truly a lock, then it
should,
> > in which case it should be safe to change the Python plugin to
> > PARALLEL ...
> >
> > I'll try it out and get back to you.
>
> Yeah, I would not be surprised if we could make the Python plugin more
> performant, but matching our glue code to the python docs for embedding
> with C code is  not trivial, so I haven't spent the time trying.
Using multiple threads is easy when you call C from pyhton. I'm not sure
how it can work when you embed python into C.

When the python pwrite() is called, the C extension is holding the GIL.
When pwrite call socket.send() python will release the GIL.

Theoretically, the thread that called pwrite is blocked and another thread
can call Python C API, but it may also break horribly if the blocked thread
modified some global state and is not expecting other code to run until the
python code returns. I never tried to do this.

A better way is to separate the management code (using oVirt python SDK)
from the transfer code (using http client). The management code should run
before the transfer, setting up the transfer, and after the transfer
(finalling it).

The transfer code can be written in C using libcurl using parallel mode.

> > Also NBD lets you multiplex commands on a single connection (which
> > does not require multi-conn or --shared).
> >
> > BTW I found that multi-conn is a big win with the Linux kernel NBD
> > client.
> >
> >> We use 4 connections by default, giving about 100% speed up
compared
> >> with one connection. 2 connections give about 80% speed up.  If
the
> >> number of connections is related to the number of coroutines, you
> >> can use -m 4 to use 4 coroutines.
> >>
> >> Using -W will improve performance. In this mode every coroutine
will
> >> do the I/O when it is ready, instead of waiting for other
coroutines
> >> and submit the I/O in the right order.
> >
> > I think Eric might have a better idea about what -m and -W really do
> > for qemu NBD client.  Maybe improve multiplexing?  They don't
enable
> > multi-conn :-(
>
> Correct.  Using -W doesn't make sense without -m (if you only have one
> worker, you might as well proceed linearly than trying to randomize
> access, but even when you have multiple threads, there are cases where
> linear operations are still useful, such as 'nbdkit streaming'. 
But -m
> is definitely the knob that controls how many outstanding I/O requests
> qemu-img is willing to use; and once you are using -m, using -W makes
> life easier for those coroutines to stay active.  The default -m1 says
> that at most one request is outstanding, so parallelism in the server is
> not utilized.  With higher -m, qemu-img issues up to that many requests
> without waiting for server answers, but all on the same NBD connection.
> Ideally, you'll get the maximum behavior as 'qemu-img -m' and
'nbdkit
> --threads' choose the same values; if either side has fewer in-flight
> operations permitted than the other, then that side has the potential to
> become a bottleneck.  Right now, nbdkit defaults to 16 threads (that is,
> up to 16 in-flight operations) for any PARALLEL plugin.
>
> And someday, I'd love to improve nbdkit's PARALLEL mode to make its
> thread-pool more of an on-demand setup (right now, we pre-create all 16
> threads up front, even if the client never reaches 16 in-flight
> operations at once, which is a bit wasteful), but other than potential
> performance improvements, it should be a transparent change to both
> plugins and clients.
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3226
> Virtualization:  qemu.org | libvirt.org
>

On Wed, Aug 5, 2020 at 5:18 PM Nir Soffer <nsoffer@redhat.com>
wrote:
>
> On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> >
> > On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
> > > I see, can change the python plugin to support multiple
connections to imageio
> > > using SERIALIZE_REQUESTS?
> > >
> > > The GiL should not limit us since the GIL is released when you
write to
> > > imageio socket, and this is likely where the plugin spends most
of the time.
> >
> > It's an interesting question and one I'd not really considered
at all.
> > Does the Python GIL actively mutex different threads if they call into
> > Python code at the same time?
>
> Yes, only one thread can run python code at the same time, there is no
> way around this. But before calling most syscalls, python releases the GIL.
> When the syscall returns, python acquires the GIL again.
>
> But thread can be switched at any point in time, so must use locking
> in the python if you modify or access shared state.
>
> >  If it's truly a lock, then it should,
> > in which case it should be safe to change the Python plugin to
> > PARALLEL ...
>
> Using multiple threads on the same http socket will not work, you must have
> multiple sockets to imageio.
And we also have to change the rhv-upload-pluign, since we create the
disk and validate
the current hosts in open(). These operation should be done once.

So the flow will be:

1. pre check
2. prepare overlay
3. create disk
4. check if the current host can do the upload
5. run qemu-img convert
6. post (create vm or delete disk on failure)
> > I'll try it out and get back to you.
> >
> > > I'm not sure what triggers using multiple connections in
qemu-img and
> > > how it decide how many connections should be used, but we report
> > > the number of writers in OPTIONS:
> > > http://ovirt.github.io/ovirt-imageio/images.html#max_writers
> > >
> > > There is a hard limit in vdsm, because it runs qemu-nbd with
> > > --shared=8, so you should
> > > not use more than 8 connections, they will just block on qemu-nbd
forever.
> >
> > It's different for qemu NBD client and server.  Eric told me on
IRC a
> > few minutes ago that qemu NBD client does not "yet" support
> > multi-conn.  However it is implemented by qemu-nbd (the server) for
> > readonly connections.
> >
> > Note that multi-conn and --shared are (a bit) different.  Multi-conn
> > is a flag which the server can send to clients to indicate not only
> > that multiple connections are possible, but also that they are come
> > with certain guarantees:
> >
> >   bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
> >   entirely without cache, or that the cache it uses is shared among
> >   all connections to the given device. In particular, if this flag is
> >   present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
> >   be visible across all connections when the server sends its reply to
> >   that command to the client. In the absence of this flag, clients
> >   SHOULD NOT multiplex their commands over more than one connection to
> >   the export.
> >
> > “--shared” limits the number of connections that the server will
> > accept at the same time (like the nbdkit limit filter).  But it would
> > still be possible for an NBD server to accept multiple connections
> > from a single client but not be multi-conn safe.
> >
> > Also NBD lets you multiplex commands on a single connection (which
> > does not require multi-conn or --shared).
> >
> > BTW I found that multi-conn is a big win with the Linux kernel NBD
> > client.
> >
> > > We use 4 connections by default, giving about 100% speed up
compared
> > > with one connection. 2 connections give about 80% speed up.  If
the
> > > number of connections is related to the number of coroutines, you
> > > can use -m 4 to use 4 coroutines.
> > >
> > > Using -W will improve performance. In this mode every coroutine
will
> > > do the I/O when it is ready, instead of waiting for other
coroutines
> > > and submit the I/O in the right order.
> >
> > I think Eric might have a better idea about what -m and -W really do
> > for qemu NBD client.  Maybe improve multiplexing?  They don't
enable
> > multi-conn :-(
> >
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> >