Libguestfs - Jan 2021 - [PATCH] v2v: rhv-upload-plugin: Defer imageio connection

On Wed, Jan 20, 2021 at 12:44 AM Nir Soffer <nirsof at gmail.com>
wrote:
>
> When using vddk input with certain vmware version, qemu-img may spend
> lot of time getting source image extents. If getting image extents takes
> more than 60 seconds, imageio server closes the idle connection, and the
> transfer will fail on the first write with:
>
> nbdkit: python[1]: error: /var/tmp/rhvupload.0OKqWA/rhv-upload-plugin.py:
pwrite: error:
> Traceback (most recent call last):
>    File "/var/tmp/rhvupload.0OKqWA/rhv-upload-plugin.py", line
94, in wrapper
>     return func(h, *args)
>    File "/var/tmp/rhvupload.0OKqWA/rhv-upload-plugin.py", line
230, in pwrite
>     r = http.getresponse()
>    File "/usr/lib64/python3.6/http/client.py", line 1361, in
getresponse
>     response.begin()
>    File "/usr/lib64/python3.6/http/client.py", line 311, in begin
>     version, status, reason = self._read_status()
>    File "/usr/lib64/python3.6/http/client.py", line 280, in
_read_status
>     raise RemoteDisconnected("Remote end closed connection
without"
>  http.client.RemoteDisconnected: Remote end closed connection without
response
>
> Fix the issue by deferring the actual connection used for the transfer
> until qemu try to write to the server:
>
> - The first connection is created in open(). This connection is used to
>   query imageio server options and initialize the handle.
>
> - The second connection is created on the first call of the nbd
>   callbacks (pwrite, zero, ...). We store it in the handle and reuse it
>   for the rest of the transfer.
>
> Since we set inactivity_timeout to 3600 seconds, oVirt will keep the
> transfer alive for one hour. If more time is needed to get image
> extents, we need to increase the inactivity timeout.
>
> I did not test this change yet, but I tested the approach using oVirt
> upload_disk.py example script. Posting early to get feedback on this
> approach.
I cannot test this change, since building virt-v2v on Fedora 32 fails.

I tried:

git clean -dxf
./autogen.sh
make

Make fails with:

...
  OCAMLCMI cmdline.cmi
  OCAMLOPT cmdline.cmx
  OCAMLCMI v2v.cmi
  OCAMLOPT v2v.cmx
File "v2v.ml", line 94, characters 20-48:
94 |   g#set_smp (min 8 (Sysconf.nr_processors_online ()));
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound module Sysconf
Hint: Did you mean Scanf?
make[3]: *** [Makefile:2753: v2v.cmx] Error 2
make[3]: Leaving directory '/home/nsoffer/src/virt-v2v/v2v'
make[2]: *** [Makefile:2069: all] Error 2
make[2]: Leaving directory '/home/nsoffer/src/virt-v2v/v2v'
make[1]: *** [Makefile:1841: all-recursive] Error 1
make[1]: Leaving directory '/home/nsoffer/src/virt-v2v'
make: *** [Makefile:1759: all] Error 2

I could not find any build instructions in virt-v2v project.

Richard, any tips on building virt-v2v?

Nir

On Wed, Jan 20, 2021 at 01:44:47AM +0200, Nir Soffer
wrote:
> I cannot test this change, since building virt-v2v on Fedora 32 fails.
> 
> I tried:
> 
> git clean -dxf
> ./autogen.sh
> make
> 
> Make fails with:
> 
> ...
>   OCAMLCMI cmdline.cmi
>   OCAMLOPT cmdline.cmx
>   OCAMLCMI v2v.cmi
>   OCAMLOPT v2v.cmx
> File "v2v.ml", line 94, characters 20-48:
> 94 |   g#set_smp (min 8 (Sysconf.nr_processors_online ()));
>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Error: Unbound module Sysconf
> Hint: Did you mean Scanf?
I think this means the common/ submodule isn't updated, so you have
to do:

git submodule update

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org