Libguestfs - Nov 2019 - [PATCH v2 1/3] rhv-upload: Log params for easier debugging

Add support for qcow2 disk format, enabled by imageio NBD backend in RHV 4.3.
To use this feature manually, you can run virt-v2v with "-of qcow2".

Here is example run:

Source disk:

$ qemu-img info /var/tmp/fedora-30.img
image: /var/tmp/fedora-30.img
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.15 GiB

virt-v2v:

$ ./run virt-v2v \
    -v \
    -i disk /var/tmp/fedora-30.img \
    -o rhv-upload \
    -oc https://engine/ovirt-engine/api \
    -op /var/tmp/engine-password \
    -on v2v \
    -os nfs1-export2 \
    -of qcow2 \
    -oa sparse \
    -oo rhv-cafile=/etc/pki/vdsm/certs/cacert.pem \
    -oo rhv-cluster=fc30 \
    -oo rhv-direct=true

...
[  71.2] Copying disk 1/1 to qemu URI json:{ "file.driver":
"nbd", "file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" } (qcow2)
             overlay file: /home/nsoffer/src/virt-v2v/tmp/v2vovl7fefa6.qcow2
      overlay device name: sda
overlay virtual disk size: 6442450944
  overlay source qemu URI: /var/tmp/fedora-30.img
    target estimated size: 2456662831
       target actual size: None

          target file: [qemu] json:{ "file.driver": "nbd",
"file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" }
        target format: qcow2
...
qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw' '-S' '64k'
'/home/nsoffer/src/virt-v2v/tmp/v2vovl7fefa6.qcow2' 'json:{
"file.driver": "nbd", "file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" }'
...

Target disk:

$ qemu-img info
/rhev/data-center/82a445b6-b5e0-43c3-b761-cc6ef972e163/55255570-983a-4d82-907a-19b964abf7ed/images/f5325b5a-22e4-4a80-ad15-f68378c42dde/ab4587de-3ddf-4178-8cc0-1a09449c970d
image:
/rhev/data-center/82a445b6-b5e0-43c3-b761-cc6ef972e163/55255570-983a-4d82-907a-19b964abf7ed/images/f5325b5a-22e4-4a80-ad15-f68378c42dde/ab4587de-3ddf-4178-8cc0-1a09449c970d
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.17 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Nir Soffer (3):
  rhv-upload: Log params for easier debugging
  v2v: Add output#tranfer_format method
  rhv-upload: Support qcow2 disk format

 v2v/output_rhv_upload.ml |  7 ++++---
 v2v/rhv-upload-plugin.py | 20 ++++++++++++++++++++
 v2v/types.ml             |  1 +
 v2v/types.mli            |  4 ++++
 v2v/v2v.ml               |  2 +-
 5 files changed, 30 insertions(+), 4 deletions(-)

-- 
2.21.0

Log the params dict passed from v2v to the plugin for easier debugging.

Here is an example output:

using params: {'diskid_file':
'/home/nsoffer/src/virt-v2v/tmp/rhvupload.To9y8J/diskid.0',
'disk_size': 6442450944, 'disk_format': 'raw',
'disk_name': 'v2v-000', 'output_name': 'v2v',
'verbose': True, 'output_conn':
'https://engine/ovirt-engine/api', 'output_password':
'/var/tmp/engine-password',
'output_storage': 'nfs1-export2', 'output_sparse': True,
'rhv_cafile': '/etc/pki/vdsm/certs/cacert.pem',
'rhv_cluster': 'fc30', 'rhv_direct': True,
'insecure': True}
---
 v2v/rhv-upload-plugin.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
index 7ed521f3..a9e82250 100644
--- a/v2v/rhv-upload-plugin.py
+++ b/v2v/rhv-upload-plugin.py
@@ -46,6 +46,7 @@ def config(key, value):
     if key == "params":
         with builtins.open(value, 'r') as fp:
             params = json.load(fp)
+        debug("using params: %s" % params)
     else:
         raise RuntimeError("unknown configuration key '%s'" %
key)
 
-- 
2.21.0

Typically the transfer format is the same as the target format. But some
outputs may want to override the transfer format but create disks using
target format.

This change does not affect existing outputs, but will using raw
transfer format in rhv_upload when using qcow2 target format.
---
 v2v/types.ml  | 1 +
 v2v/types.mli | 4 ++++
 v2v/v2v.ml    | 2 +-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/v2v/types.ml b/v2v/types.ml
index 977d9bda..8de48aec 100644
--- a/v2v/types.ml
+++ b/v2v/types.ml
@@ -529,6 +529,7 @@ class virtual output = object
   method virtual supported_firmware : target_firmware list
   method check_target_firmware (_ : guestcaps) (_ : target_firmware) = ()
   method override_output_format (_ : overlay) = (None : string option)
+  method transfer_format t = t.target_format
   method virtual prepare_targets : string -> (string * overlay) list ->
target_file list
   method disk_create = (open_guestfs ())#disk_create
   method disk_copied (_ : target) (_ : int) (_ : int) = ()
diff --git a/v2v/types.mli b/v2v/types.mli
index 4c1d3595..9ceee8a2 100644
--- a/v2v/types.mli
+++ b/v2v/types.mli
@@ -488,6 +488,10 @@ class virtual output : object
       line (silently).  It’s best not to do this, instead modify
       prepare_targets so it gives an error if the output format
       chosen is not supported by the target. *)
+  method transfer_format : target -> string
+  (** Typically the transfer format is same as the target format, but
+      some outputs may need to overide the transfer format, but create disk
+      using target format. *)
   method virtual prepare_targets : string -> (string * overlay) list ->
target_file list
   (** Called after conversion but before copying to prepare (but {b not}
       create) the target file.  The [(string * overlay list)] parameter
diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 03590c9e..f9d81460 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -739,7 +739,7 @@ and copy_targets cmdline targets input output            |
TargetURI uri -> uri in
         [ "qemu-img"; "convert" ] @
         (if not (quiet ()) then [ "-p" ] else []) @
-        [ "-n"; "-f"; "qcow2"; "-O";
t.target_format ] @
+        [ "-n"; "-f"; "qcow2"; "-O";
output#transfer_format t ] @
         (if cmdline.compressed then [ "-c" ] else []) @
         [ "-S"; "64k" ] @
         [ overlay_file; filename ] in
-- 
2.21.0

When using oVirt >= 4.3, we can enable the NBD based backend in imageio
by specifying that we transfer raw data when creating a transfer. With  
the NBD backend, we can import to disks using qcow2 format.

To make it work, we override output#transfer_format to return always raw
format, but we create the disk on RHV side using qcow2 format.

The pipeline looks like this:

    qemu-img convert <nbd> rhv-upload <http> imageio <nbd>
qemu-nbd
---
 v2v/output_rhv_upload.ml |  7 ++++---
 v2v/rhv-upload-plugin.py | 19 +++++++++++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml
index 3514714b..7b5ad7a8 100644
--- a/v2v/output_rhv_upload.ml
+++ b/v2v/output_rhv_upload.ml
@@ -174,7 +174,7 @@ See also the virt-v2v-output-rhv(1) manual.")
       error (f_"nbdkit was compiled without SELinux support.  You will
have to recompile nbdkit with libselinux-devel installed, or else set SELinux to
Permissive mode while doing the conversion.")
   in
 
-  (* Output format/sparse must be raw/sparse.  We may be able to
+  (* Output sparse must be sparse.  We may be able to
    * lift this limitation in future, but it requires changes on the
    * RHV side.  See TODO file for details.  XXX
    *)
@@ -287,6 +287,8 @@ object
 
   method supported_firmware = [ TargetBIOS; TargetUEFI ]
 
+  method transfer_format t = "raw"
+
   (* rhev-apt.exe will be installed (if available). *)
   method install_rhev_apt = true
 
@@ -333,8 +335,7 @@ object
         let disk_format            match target_format with
           | "raw" as fmt -> fmt
-          | "qcow2" ->
-             error_current_limitation "-of raw"
+          | "qcow2" as fmt -> fmt
           | _ ->
              error (f_"rhv-upload: -of %s: Only output format ‘raw’ or
‘qcow2’ is supported.  If the input is in a different format then force one of
these output formats by adding either ‘-of raw’ or ‘-of qcow2’ on the command
line.")
                    target_format in
diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
index a9e82250..75e4f404 100644
--- a/v2v/rhv-upload-plugin.py
+++ b/v2v/rhv-upload-plugin.py
@@ -18,6 +18,7 @@
 
 import builtins
 import functools
+import inspect
 import json
 import logging
 import socket
@@ -448,6 +449,8 @@ def create_disk(connection):
             name = params['disk_name'],
             description = "Uploaded by virt-v2v",
             format = disk_format,
+            # XXX For qcow2 disk on block storage, we should use the estimated
+            # size, based on qemu-img measure of the overlay.
             initial_size = params['disk_size'],
             provisioned_size = params['disk_size'],
             # XXX Ignores params['output_sparse'].
@@ -489,11 +492,16 @@ def create_transfer(connection, disk, host):
     system_service = connection.system_service()
     transfers_service = system_service.image_transfers_service()
 
+    extra = {}
+    if transfer_supports_format():
+        extra["format"] = types.DiskFormat.RAW
+
     transfer = transfers_service.add(
         types.ImageTransfer(
             disk = types.Disk(id = disk.id),
             host = host,
             inactivity_timeout = 3600,
+            **extra,
         )
     )
 
@@ -591,6 +599,17 @@ def finalize_transfer(connection, transfer, disk_id):
                 "timed out waiting for transfer %s to finalize"
                 % transfer.id)
 
+def transfer_supports_format():
+    """
+    Return True if transfer supports the "format" argument, enabing
the NBD
+    bakend on imageio side, which allows uploading to qcow2 images.
+
+    This feature was added in ovirt 4.3. We assume that the SDK version matches
+    engine version.
+    """
+    sig = inspect.signature(types.ImageTransfer)
+    return "format" in sig.parameters
+
 # oVirt imageio operations
 
 def parse_transfer_url(transfer):
-- 
2.21.0

On Tue, Nov 26, 2019 at 05:07:30AM +0200, Nir Soffer
wrote:
> Log the params dict passed from v2v to the plugin for easier debugging.
> 
> Here is an example output:
> 
> using params: {'diskid_file':
'/home/nsoffer/src/virt-v2v/tmp/rhvupload.To9y8J/diskid.0',
> 'disk_size': 6442450944, 'disk_format': 'raw',
'disk_name': 'v2v-000', 'output_name': 'v2v',
> 'verbose': True, 'output_conn':
'https://engine/ovirt-engine/api', 'output_password':
'/var/tmp/engine-password',
I have verified that this only logs the file containing the password,
not the actual password.
> 'output_storage': 'nfs1-export2', 'output_sparse':
True, 'rhv_cafile': '/etc/pki/vdsm/certs/cacert.pem',
> 'rhv_cluster': 'fc30', 'rhv_direct': True,
'insecure': True}
> ---
>  v2v/rhv-upload-plugin.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
> index 7ed521f3..a9e82250 100644
> --- a/v2v/rhv-upload-plugin.py
> +++ b/v2v/rhv-upload-plugin.py
> @@ -46,6 +46,7 @@ def config(key, value):
>      if key == "params":
>          with builtins.open(value, 'r') as fp:
>              params = json.load(fp)
> +        debug("using params: %s" % params)
>      else:
>          raise RuntimeError("unknown configuration key
'%s'" % key)
>  
ACK

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Seems reasonable, ACK series.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

I pushed this series as:
https://github.com/libguestfs/virt-v2v/commit/48455a9185543bd6273fa881184caa10f01b606f

On Tue, Nov 26, 2019 at 5:07 AM Nir Soffer <nirsof@gmail.com>
wrote:
>
> Add support for qcow2 disk format, enabled by imageio NBD backend in RHV
4.3.
> To use this feature manually, you can run virt-v2v with "-of
qcow2".
>
> Here is example run:
>
> Source disk:
>
> $ qemu-img info /var/tmp/fedora-30.img
> image: /var/tmp/fedora-30.img
> file format: raw
> virtual size: 6 GiB (6442450944 bytes)
> disk size: 1.15 GiB
>
> virt-v2v:
>
> $ ./run virt-v2v \
>     -v \
>     -i disk /var/tmp/fedora-30.img \
>     -o rhv-upload \
>     -oc https://engine/ovirt-engine/api \
>     -op /var/tmp/engine-password \
>     -on v2v \
>     -os nfs1-export2 \
>     -of qcow2 \
>     -oa sparse \
>     -oo rhv-cafile=/etc/pki/vdsm/certs/cacert.pem \
>     -oo rhv-cluster=fc30 \
>     -oo rhv-direct=true
>
> ...
> [  71.2] Copying disk 1/1 to qemu URI json:{ "file.driver":
"nbd", "file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" } (qcow2)
>              overlay file:
/home/nsoffer/src/virt-v2v/tmp/v2vovl7fefa6.qcow2
>       overlay device name: sda
> overlay virtual disk size: 6442450944
>   overlay source qemu URI: /var/tmp/fedora-30.img
>     target estimated size: 2456662831
>        target actual size: None
>
>           target file: [qemu] json:{ "file.driver":
"nbd", "file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" }
>         target format: qcow2
> ...
> qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw' '-S' '64k'
'/home/nsoffer/src/virt-v2v/tmp/v2vovl7fefa6.qcow2' 'json:{
"file.driver": "nbd", "file.path":
"/home/nsoffer/src/virt-v2v/tmp/rhvupload.6EmoNE/nbdkit0.sock",
"file.export": "/" }'
> ...
>
> Target disk:
>
> $ qemu-img info
/rhev/data-center/82a445b6-b5e0-43c3-b761-cc6ef972e163/55255570-983a-4d82-907a-19b964abf7ed/images/f5325b5a-22e4-4a80-ad15-f68378c42dde/ab4587de-3ddf-4178-8cc0-1a09449c970d
> image:
/rhev/data-center/82a445b6-b5e0-43c3-b761-cc6ef972e163/55255570-983a-4d82-907a-19b964abf7ed/images/f5325b5a-22e4-4a80-ad15-f68378c42dde/ab4587de-3ddf-4178-8cc0-1a09449c970d
> file format: qcow2
> virtual size: 6 GiB (6442450944 bytes)
> disk size: 1.17 GiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
>
> Nir Soffer (3):
>   rhv-upload: Log params for easier debugging
>   v2v: Add output#tranfer_format method
>   rhv-upload: Support qcow2 disk format
>
>  v2v/output_rhv_upload.ml |  7 ++++---
>  v2v/rhv-upload-plugin.py | 20 ++++++++++++++++++++
>  v2v/types.ml             |  1 +
>  v2v/types.mli            |  4 ++++
>  v2v/v2v.ml               |  2 +-
>  5 files changed, 30 insertions(+), 4 deletions(-)
>
> --
> 2.21.0
>