Libguestfs - Aug 2021 - [PATCH virt-v2v] v2v: -o rhv-upload: Enable multi-conn

On Mon, Aug 2, 2021 at 10:41 AM Richard W.M. Jones <rjones at redhat.com>
wrote:
>
> Hi Nir,
>
> The experimental patch attached (for discussion) would enable
> multi-conn.  For current virt-v2v this would have no effect since
> "qemu-img convert" does not make multi-conn NBD connections.
>
> However for modular virt-v2v[1] which uses nbdcopy this would allow
> nbdcopy to make multiple NBD connections to nbdkit running the plugin.
> (By default 4 connections).
>
> If I understand the rhv-upload-plugin code correctly, there is a pool
> per NBD connection at the moment, so this would create 4 pools and 4*4
> backend connections to imageio.  This is possibly not desirable, but I
> guess we can turn the pool into a global object.  (Note each NBD
> connection has the same (destination_url, host, options) tuple).
Creating 4*4 connections will not work since on imageio side we have qem-nbd
supporting up to 8 connections. What will happen is that http request
will block once
there are 8 connected client on imageio side. The http client will
connect only when
some other clients have disconnected, which may never happen, so this
will likely
end with a deadlock.

If we have multi_conn we don't need the pool, we can have a configuration
value
to control the number of connections, and use single connection (that
looks like a pool)
in this case.

I'm not about how multi_con works in the python plugin - do we get one
open() call
or 4 open() calls?

If we get 4 open() calls, this cannot work since we mix control flow
and data flow in the
rhv plugin. Each open call will try to create a new disk and every
connection will upload
to a different disk.

multi_con can work if we have:

- single open() call
- many write/zero/flush calls
- single close() call

If we have:

- one open() for nbd connection
- many write/zero calls per connection
- one close() for nbd connection

We need to separate the control flow stuff (e.g. create disk) from the
plugin, and do it in
another step of the process.
> Naturally I've not actually tested any of this.  There are some
> advantages from the nbdcopy side of things, especially because it is
> able to query extents on the input side in parallel which seems to be
> advantageous for inputs with slow extent querying (like VDDK).  There
> may be further advantages on the output side because it would allow us
> to write data to the RHV upload plugin over four sockets, allowing
> better use of multiple cores.
It may work better when using HTTPS, the TLS part is implemented in C
and can run in parallel.

Nir

On Mon, Aug 02, 2021 at 03:35:36PM +0300, Nir Soffer
wrote:
> I'm not about how multi_con works in the python plugin - do we get one
> open() call
> or 4 open() calls?
The NBD client is supposed to:

 - make a single NBD connection

 - test the multi-conn flag on that connection

 - if it is true and the client wants to proceed with multi-conn,
   make N-1 further NBD connections

 - each NBD connection operates separately, except for the multi-conn
   guarantees related to accurate flushing and write tearing:

   * if you flush or FUA on one connection, before you return on that
     connection, any caches in the other connections must also be
     flushed

nbdcopy:

 - correctly implements the NBD protocol and multi-conn (or if it
   doesn't, it's a bug!)

 - makes large enough requests that write tearing is unlikely to
   be an issue

 - doesn't use FUA

 - issues separate flush calls on every connection at the end

So there will be multiple open() calls in the Python plugin.
> If we get 4 open() calls, this cannot work since we mix control flow
> and data flow in the
> rhv plugin. Each open call will try to create a new disk and every
> connection will upload
> to a different disk.
Interesting.

I did try it now.  It's actually slower :-(  But I didn't change how
HTTP pools worked.

However it didn't appear to create multiple disks, unless it only
attached one and the others are hidden somehow.  I'll PM you the admin
details for my RHV cluster so you can take a look.

It has to be said this assumption that open() creates the disk is not
a good one.  It's certainly not guaranteed that even with "qemu-img
convert" only a single NBD connection will ever be opened.  It's quite
valid for an NBD client to open a connection and query properties
about it, and then perhaps close it and open another one.  So I guess
we have to fix this anyway.
> multi_con can work if we have:
> 
> - single open() call
> - many write/zero/flush calls
> - single close() call
> 
> If we have:
> 
> - one open() for nbd connection
> - many write/zero calls per connection
> - one close() for nbd connection
> 
> We need to separate the control flow stuff (e.g. create disk) from the
> plugin, and do it in
> another step of the process.
> 
> > Naturally I've not actually tested any of this.  There are some
> > advantages from the nbdcopy side of things, especially because it is
> > able to query extents on the input side in parallel which seems to be
> > advantageous for inputs with slow extent querying (like VDDK).  There
> > may be further advantages on the output side because it would allow us
> > to write data to the RHV upload plugin over four sockets, allowing
> > better use of multiple cores.
> 
> It may work better when using HTTPS, the TLS part is implemented in C
> and can run in parallel.
Yes, or even with kTLS it may run in multiple kernel threads.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW