Libguestfs - Feb 2017 - Re: [PATCH v8 2/4] v2v: add function qemu_img_version to utils

v8:
- split the big patch into several commits

v7:
- rebased because patch 1/3 has been pushed
- changes to nsplit have been dropped (2/3)
- addressed Richard's comments, notably the subfolder function was moved to
  mllib and renamed to subdirectory

v6:
- just rebase

v5:
- rebase, patches 1,3,5 were merged
- 1/3: we still need to discuss whether to detect compressed discs
- 2/3:
  - renamed argument noempty to keep_empty
  - tests were not run
- 3/3: 
  - using JSON module to generate JSON (as suggested by Pino)
  - all the other comments raised by Pino


v4:
- rebase to more recent master
- 1/6: using just "quote" instead of "Filename.quote"
- 2/6: reformated block of code according to Richards suggestion
- 4/6: added tests for nsplit


v3: Addressed Pino's comments, namely:
- input_ova.ml
  - untar takes list of paths
  - renamed untar_partial to untar_metadata
  - replaced uggly regex with nsplit
- tests
  - test changes are part of the main commit
  - renamed test-data/guestfs-hashsums.sh to test-data/test-utils.sh
  - renamed qemu_version to qemu_is_version and moved it to
    test-data/test-utils.sh
  - normalize paths in expect files


v2:
- rewritten the tar invocations, the output processing is now done in
  OcaML rather than with a shell code; it turned out to be easier and
  more readable than I have feared.
- added QEMU version check
- addressed Pino's comments
- changed tests; the expected result is now based on the QEMU used
  during testing

This series is related to the problem of inefficient import of OVA
files. The needed enhancements of QEMU were merged into the codebase and
are available in QEMU 2.8. From there we can use 'size' and
'offset' options in raw driver to tell QEMU to use only subset of a file
as an image.

The patch set is more or less complete. The only outstanding issue is
the missing detection of sparse files inside tar. But this can be done
in a separate patch. As pointed out before I didn't find a way how to
detect that by using the tar tool only and would probably require use of
some external library.


Tomáš Golembiovský (4):
  mllib: add subdirectory function to common_utils
  v2v: add function qemu_img_version to utils
  v2v: add function find_file_in_tar to utils
  v2v: ova: don't extract files from OVA if it's not needed

 mllib/common_utils.ml                   |   9 +++
 mllib/common_utils.mli                  |  10 +++
 mllib/common_utils_tests.ml             |   7 ++
 test-data/test-utils.sh                 |  19 +++++
 v2v/Makefile.am                         |   2 +
 v2v/input_ova.ml                        | 109 +++++++++++++++++++++----
 v2v/test-v2v-i-ova-formats.sh           |   5 +-
 v2v/test-v2v-i-ova-subfolders.expected2 |  18 +++++
 v2v/test-v2v-i-ova-subfolders.sh        |  13 ++-
 v2v/test-v2v-i-ova-tar.expected         |  18 +++++
 v2v/test-v2v-i-ova-tar.expected2        |  18 +++++
 v2v/test-v2v-i-ova-tar.ovf              | 138 ++++++++++++++++++++++++++++++++
 v2v/test-v2v-i-ova-tar.sh               |  72 +++++++++++++++++
 v2v/test-v2v-i-ova-two-disks.expected2  |  19 +++++
 v2v/test-v2v-i-ova-two-disks.sh         |  13 ++-
 v2v/utils.ml                            |  72 +++++++++++++++++
 v2v/utils.mli                           |  12 +++
 17 files changed, 530 insertions(+), 24 deletions(-)
 create mode 100644 v2v/test-v2v-i-ova-subfolders.expected2
 create mode 100644 v2v/test-v2v-i-ova-tar.expected
 create mode 100644 v2v/test-v2v-i-ova-tar.expected2
 create mode 100644 v2v/test-v2v-i-ova-tar.ovf
 create mode 100755 v2v/test-v2v-i-ova-tar.sh
 create mode 100644 v2v/test-v2v-i-ova-two-disks.expected2

-- 
2.11.0

Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
---
 mllib/common_utils.ml       |  9 +++++++++
 mllib/common_utils.mli      | 10 ++++++++++
 mllib/common_utils_tests.ml |  7 +++++++
 3 files changed, 26 insertions(+)

diff --git a/mllib/common_utils.ml b/mllib/common_utils.ml
index e9ae6a4a2..a79abdd7e 100644
--- a/mllib/common_utils.ml
+++ b/mllib/common_utils.ml
@@ -236,6 +236,15 @@ end
 let (//) = Filename.concat
 let quote = Filename.quote
 
+let subdirectory parent path +  if path = parent then
+    ""
+  else if String.is_prefix path (parent // "") then
+    let len = String.length parent in
+    String.sub path (len+1) (String.length path - len-1)
+  else
+    raise (Invalid_argument (sprintf "%S is not a path prefix of %S"
parent path))
+
 let ( +^ ) = Int64.add
 let ( -^ ) = Int64.sub
 let ( *^ ) = Int64.mul
diff --git a/mllib/common_utils.mli b/mllib/common_utils.mli
index 722e528e5..977ce6576 100644
--- a/mllib/common_utils.mli
+++ b/mllib/common_utils.mli
@@ -116,6 +116,16 @@ val ( // ) : string -> string -> string
 val quote : string -> string
 (** Shell-safe quoting of a string (alias for {!Filename.quote}). *)
 
+val subdirectory : string -> string -> string
+(** [subdirectory parent path] returns subdirectory part of [path] relative
+    to the [parent]. If [path] and [parent] point to the same directory empty
+    string is returned.
+
+    Note: path normalization on arguments is NOT performed!
+
+    If [parent] is not a path prefix of [path] the function raises
+    [Invalid_argument]. *)
+
 val ( +^ ) : int64 -> int64 -> int64
 val ( -^ ) : int64 -> int64 -> int64
 val ( *^ ) : int64 -> int64 -> int64
diff --git a/mllib/common_utils_tests.ml b/mllib/common_utils_tests.ml
index 77b0524c1..aacc01e04 100644
--- a/mllib/common_utils_tests.ml
+++ b/mllib/common_utils_tests.ml
@@ -27,6 +27,12 @@ let assert_equal_int = assert_equal ~printer:(fun x ->
string_of_int x)
 let assert_equal_int64 = assert_equal ~printer:(fun x -> Int64.to_string x)
 let assert_equal_stringlist = assert_equal ~printer:(fun x -> "("
^ (String.escaped (String.concat "," x)) ^ ")")
 
+let test_subdirectory ctx +  assert_equal_string "" (subdirectory
"/foo" "/foo");
+  assert_equal_string "" (subdirectory "/foo"
"/foo/");
+  assert_equal_string "bar" (subdirectory "/foo"
"/foo/bar");
+  assert_equal_string "bar/baz" (subdirectory "/foo"
"/foo/bar/baz")
+
 (* Test Common_utils.int_of_le32 and Common_utils.le32_of_int. *)
 let test_le32 ctx    assert_equal_int64 0x20406080L (int_of_le32
"\x80\x60\x40\x20");
@@ -129,6 +135,7 @@ let test_string_lines_split ctx  let suite    "mllib
Common_utils" >:::
     [
+      "subdirectory" >:: test_subdirectory;
       "numeric.le32" >:: test_le32;
       "sizes.parse_resize" >:: test_parse_resize;
       "sizes.human_size" >:: test_human_size;
-- 
2.11.0

The function returns version of qemu-img as a tuple (major, minor), or
the value (0,9) in case there was an error detecting the version.

Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
---
 v2v/utils.ml  | 21 +++++++++++++++++++++
 v2v/utils.mli |  5 +++++
 2 files changed, 26 insertions(+)

diff --git a/v2v/utils.ml b/v2v/utils.ml
index 17ad8a29c..896e16004 100644
--- a/v2v/utils.ml
+++ b/v2v/utils.ml
@@ -90,3 +90,24 @@ let du filename    match lines with
   | line::_ -> Int64.of_string line
   | [] -> invalid_arg filename
+
+let qemu_img_version () +  let lines = external_command "qemu-img
--version" in
+  match lines with
+  | [] -> error ("'qemu-img --version' returned no
output")
+  | line :: _ ->
+      let rex = Str.regexp
+        "qemu-img version \\([0-9]+\\)\\.\\([0-9]+\\)" in
+      if Str.string_match rex line 0 then (
+        try
+          int_of_string (Str.matched_group 1 line),
+          int_of_string (Str.matched_group 2 line)
+        with Failure _ ->
+          warning (f_"failed to parse qemu-img version(%S), assuming
0.9")
+            line;
+          0, 9
+      ) else (
+        warning (f_"failed to read qemu-img version(%S), assuming
0.9")
+          line;
+        0, 9
+      )
diff --git a/v2v/utils.mli b/v2v/utils.mli
index 5eacb4aec..a7f8b6e84 100644
--- a/v2v/utils.mli
+++ b/v2v/utils.mli
@@ -50,3 +50,8 @@ val du : string -> int64
 
     This can raise either [Failure] or [Invalid_argument] in case
     of errors. *)
+
+val qemu_img_version : unit -> int * int
+(** Returns version of qemu-img as a tuple (major, minor).
+
+    In case of error (0,9) is returned. *)
-- 
2.11.0

The function looks up file in tar archive and returns a tuple containing
which at byte it starts and how long the file is.

Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
---
 v2v/utils.ml  | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 v2v/utils.mli |  7 +++++++
 2 files changed, 58 insertions(+)

diff --git a/v2v/utils.ml b/v2v/utils.ml
index 896e16004..111dc0ea9 100644
--- a/v2v/utils.ml
+++ b/v2v/utils.ml
@@ -111,3 +111,54 @@ let qemu_img_version ()            line;
         0, 9
       )
+
+let find_file_in_tar tar filename +  let lines = external_command (sprintf
"tar tRvf %s" (Filename.quote tar)) in
+  let rec loop lines +    match lines with
+    | [] -> raise Not_found
+    | line :: lines -> (
+      (* Lines have the form:
+        * block <offset>: <perms> <owner>/<group>
<size> <mdate> <mtime> <file>
+        *)
+      let elems = Str.bounded_split (Str.regexp " +") line 8 in
+      if List.length elems = 8 && List.hd elems = "block"
then (
+        let elems = Array.of_list elems in
+        let offset = elems.(1) in
+        let size = elems.(4) in
+        let fname = elems.(7) in
+
+        if fname <> filename then
+          loop lines
+        else (
+          let offset +            try
+              (* There should be a colon at the end *)
+              let i = String.rindex offset ':' in
+              if i == (String.length offset)-1 then
+                Int64.of_string (String.sub offset 0 i)
+              else
+                raise (Failure "colon at wrong position")
+            with Failure _ | Not_found ->
+              raise (Failure (sprintf "invalid offset returned by tar:
%S"
+                offset)) in
+
+          let size +            try Int64.of_string size
+            with Failure _ ->
+              raise (Failure (sprintf
+                "invalid size returned by tar: %S" size)) in
+
+          (* Note: Offset is actualy block number and there is a single
+            * block with tar header at the beginning of the file. So skip
+            * the header and convert the block number to bytes before
+            * returning.
+            *)
+          (offset +^ 1L) *^ 512L, size
+        )
+      ) else
+        raise (Failure (sprintf
+          "failed to parse line returned by tar: %S" line))
+    )
+  in
+  loop lines
diff --git a/v2v/utils.mli b/v2v/utils.mli
index a7f8b6e84..9c92d93be 100644
--- a/v2v/utils.mli
+++ b/v2v/utils.mli
@@ -55,3 +55,10 @@ val qemu_img_version : unit -> int * int
 (** Returns version of qemu-img as a tuple (major, minor).
 
     In case of error (0,9) is returned. *)
+
+val find_file_in_tar : string -> string -> int64 * int64
+(** [find_file_in_tar tar filename] looks up file in [tar] archive and returns
+    a tuple containing at which byte it starts and how long the file is.
+
+    Function raises [Not_found] if there is no such file inside [tar] and
+    [Failure] if there is any error parsing the tar output. *)
-- 
2.11.0

We don't have to always extract all files from the OVA archive. The OVA,
as defined in the standard, is plain tar. We can work directly over the
tar archive if we use correct 'offset' and 'size' options when
defining
the backing file for QEMU. This puts much lower requirement on available
disk space.

Since the virt-v2v behaviour for OVA input now depends on QEMU version
available this affects some of the tests. Expected result of the
affected also has to depend on the QEMU used thus such tests will have
two *.expected files.

Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
---
 test-data/test-utils.sh                 |  19 +++++
 v2v/Makefile.am                         |   2 +
 v2v/input_ova.ml                        | 109 +++++++++++++++++++++----
 v2v/test-v2v-i-ova-formats.sh           |   5 +-
 v2v/test-v2v-i-ova-subfolders.expected2 |  18 +++++
 v2v/test-v2v-i-ova-subfolders.sh        |  13 ++-
 v2v/test-v2v-i-ova-tar.expected         |  18 +++++
 v2v/test-v2v-i-ova-tar.expected2        |  18 +++++
 v2v/test-v2v-i-ova-tar.ovf              | 138 ++++++++++++++++++++++++++++++++
 v2v/test-v2v-i-ova-tar.sh               |  72 +++++++++++++++++
 v2v/test-v2v-i-ova-two-disks.expected2  |  19 +++++
 v2v/test-v2v-i-ova-two-disks.sh         |  13 ++-
 12 files changed, 420 insertions(+), 24 deletions(-)
 create mode 100644 v2v/test-v2v-i-ova-subfolders.expected2
 create mode 100644 v2v/test-v2v-i-ova-tar.expected
 create mode 100644 v2v/test-v2v-i-ova-tar.expected2
 create mode 100644 v2v/test-v2v-i-ova-tar.ovf
 create mode 100755 v2v/test-v2v-i-ova-tar.sh
 create mode 100644 v2v/test-v2v-i-ova-two-disks.expected2

diff --git a/test-data/test-utils.sh b/test-data/test-utils.sh
index 86a5aaf12..1c4abe392 100755
--- a/test-data/test-utils.sh
+++ b/test-data/test-utils.sh
@@ -54,3 +54,22 @@ do_sha256 ()
       ;;
   esac
 }
+
+# Returns 0 if QEMU version is greater or equal to the arguments
+qemu_is_version() {
+    if [ $# -ne 2 ] ; then
+        echo "Usage: $0 <major_version> <minor_version>"
>&2
+        return 3
+    fi
+
+
+    [[ "$(qemu-img --version)" =~ 'qemu-img version
'([0-9]+)\.([0-9]+) ]] || return 2
+    QMAJ=${BASH_REMATCH[1]}
+    QMIN=${BASH_REMATCH[2]}
+
+    if [ \( $QMAJ -gt $1 \) -o \( $QMAJ -eq $1 -a $QMIN -ge $2 \) ] ; then
+        return 0
+    fi
+
+    return 1
+}
diff --git a/v2v/Makefile.am b/v2v/Makefile.am
index 9189aaf12..d62ac477e 100644
--- a/v2v/Makefile.am
+++ b/v2v/Makefile.am
@@ -260,6 +260,7 @@ TESTS_ENVIRONMENT = $(top_builddir)/run --test
 
 TESTS = \
 	test-v2v-docs.sh \
+	test-v2v-i-ova-tar.sh \
 	test-v2v-i-ova-formats.sh \
 	test-v2v-i-ova-gz.sh \
 	test-v2v-i-ova-subfolders.sh \
@@ -359,6 +360,7 @@ EXTRA_DIST += \
 	test-v2v-i-ova-subfolders.expected \
 	test-v2v-i-ova-subfolders.ovf \
 	test-v2v-i-ova-subfolders.sh \
+	test-v2v-i-ova-tar.sh \
 	test-v2v-i-ova-two-disks.expected \
 	test-v2v-i-ova-two-disks.ovf \
 	test-v2v-i-ova-two-disks.sh \
diff --git a/v2v/input_ova.ml b/v2v/input_ova.ml
index 40f723633..d2b48c180 100644
--- a/v2v/input_ova.ml
+++ b/v2v/input_ova.ml
@@ -39,17 +39,23 @@ object
 
   method source ()  
-    let untar ?(format = "") file outdir -      let cmd = [
"tar"; sprintf "-x%sf" format; file; "-C"; outdir
] in
+    (* Untar part or all files from tar archive. If [paths] is specified it is
+     * a list of paths in the tar archive.
+     *)
+    let untar ?(format = "") ?paths file outdir +      let cmd +     
[ "tar"; sprintf "-x%sf" format; file; "-C";
outdir ]
+        @ match paths with None -> [] | Some p -> p in
       if run_command cmd <> 0 then
-        error (f_"error unpacking %s, see earlier error messages")
ova in
+        error (f_"error unpacking %s, see earlier error messages")
ova
+    in
 
     (* Extract ova file. *)
-    let exploded +    let exploded, partial        (* The spec allows a
directory to be specified as an ova.  This
        * is also pretty convenient.
        *)
-      if is_directory ova then ova
+      if is_directory ova then ova, false
       else (
         let uncompress_head zcat file            let cmd = sprintf "%s
%s" zcat (quote file) in
@@ -67,11 +73,35 @@ object
 
           tmpfile in
 
+        (* Untar only ovf and manifest from the archive *)
+        let untar_metadata ova outdir +          let files +           
external_command (sprintf "tar -tf %s" (Filename.quote ova)) in
+          let files +            filter_map (fun f ->
+              if Filename.check_suffix f ".ovf" ||
+                  Filename.check_suffix f ".mf" then
+                Some f
+              else None
+            ) files in
+          untar ~paths:files ova outdir
+        in
+
         match detect_file_type ova with
         | `Tar ->
           (* Normal ovas are tar file (not compressed). *)
-          untar ova tmpdir;
-          tmpdir
+          let qmajor, qminor = qemu_img_version () in
+          if qmajor > 2 || (qmajor == 2 && qminor >= 8) then (
+            (* If QEMU is 2.8 or newer we don't have to extract everything.
+             * We can access disks inside the tar archive directly.
+             *)
+            untar_metadata ova tmpdir;
+            tmpdir, true
+          ) else (
+            untar ova tmpdir;
+            tmpdir, false
+          )
+
         | `Zip ->
           (* However, although not permitted by the spec, people ship
            * zip files as ova too.
@@ -81,7 +111,7 @@ object
             [ "-j"; "-d"; tmpdir; ova ] in
           if run_command cmd <> 0 then
             error (f_"error unpacking %s, see earlier error
messages") ova;
-          tmpdir
+          tmpdir, false
         | (`GZip|`XZ) as format ->
           let zcat, tar_fmt              match format with
@@ -94,7 +124,7 @@ object
           (match tmpfiletype with
           | `Tar ->
             untar ~format:tar_fmt ova tmpdir;
-            tmpdir
+            tmpdir, false
           | `Zip | `GZip | `XZ | `Unknown ->
             error (f_"%s: unsupported file format\n\nFormats which we
currently understand for '-i ova' are: tar (uncompressed, compress with
gzip or xz), zip") ova
           )
@@ -152,6 +182,7 @@ object
       fun mf ->
         debug "processing manifest %s" mf;
         let mf_folder = Filename.dirname mf in
+        let mf_subfolder = subdirectory exploded mf_folder in
         let chan = open_in mf in
         let rec loop ()            let line = input_line chan in
@@ -160,7 +191,11 @@ object
             let disk = Str.matched_group 2 line in
             let expected = Str.matched_group 3 line in
             let csum = Checksums.of_string mode expected in
-            try Checksums.verify_checksum csum (mf_folder // disk)
+            try
+              if partial then
+                Checksums.verify_checksum csum ~tar:ova (mf_subfolder // disk)
+              else
+                Checksums.verify_checksum csum (mf_folder // disk)
             with Checksums.Mismatched_checksum (_, actual) ->
               error (f_"checksum of disk %s does not match manifest %s
(actual %s(%s) = %s, expected %s(%s) = %s)")
                 disk mf mode disk actual mode disk expected;
@@ -283,9 +318,25 @@ object
             | Some "gzip" -> true
             | Some s -> error (f_"unsupported compression in OVF:
%s") s in
 
-          (* Does the file exist and is it readable? *)
-          let filename = ovf_folder // filename in
-          Unix.access filename [Unix.R_OK];
+          let partial +            if compressed && partial then (
+              (* We cannot access compressed disk inside the tar; we have to
+               * extract it *)
+              untar ~paths:[(subdirectory exploded ovf_folder) // filename]
+                ova tmpdir;
+              false
+            )
+            else
+              partial in
+
+          let filename +            if partial then
+              (subdirectory exploded ovf_folder) // filename
+            else (
+              (* Does the file exist and is it readable? *)
+              Unix.access (ovf_folder // filename) [Unix.R_OK];
+              ovf_folder // filename
+            ) in
 
           (* The spec allows the file to be gzip-compressed, in which case
            * we must uncompress it into the tmpdir.
@@ -302,9 +353,39 @@ object
             )
             else filename in
 
+          let qemu_uri +            if not partial then (
+              filename
+            )
+            else (
+              let offset, size +                try find_file_in_tar ova
filename
+                with
+                | Not_found ->
+                  error (f_"file '%s' not found in the ova")
filename
+                | Failure msg -> error (f_"%s") msg in
+              (* QEMU requires size aligned to 512 bytes. This is safe because
+               * tar also works with 512 byte blocks.
+               *)
+              let size = roundup64 size 512L in
+              let doc = [
+                "file", JSON.Dict [
+                  "driver", JSON.String "raw";
+                  "offset", JSON.Int64 offset;
+                  "size", JSON.Int64 size;
+                  "file", JSON.Dict [
+                    "filename", JSON.String ova]
+                  ]
+                ] in
+              let uri +                sprintf "json:%s"
(JSON.string_of_doc ~fmt:JSON.Compact doc) in
+              debug "json: %s" uri;
+              uri
+            ) in
+
           let disk = {
             s_disk_id = i;
-            s_qemu_uri = filename;
+            s_qemu_uri = qemu_uri;
             s_format = Some "vmdk";
             s_controller = controller;
           } in
diff --git a/v2v/test-v2v-i-ova-formats.sh b/v2v/test-v2v-i-ova-formats.sh
index bd3e30ee8..0eec600eb 100755
--- a/v2v/test-v2v-i-ova-formats.sh
+++ b/v2v/test-v2v-i-ova-formats.sh
@@ -22,7 +22,7 @@ unset CDPATH
 export LANG=C
 set -e
 
-formats="tar zip tar-gz tar-xz"
+formats="zip tar-gz tar-xz"
 
 if [ -n "$SKIP_TEST_V2V_I_OVA_FORMATS_SH" ]; then
     echo "$0: test skipped because environment variable is set"
@@ -63,9 +63,6 @@ cp ../test-v2v-i-ova-formats.ovf .
 
 for format in $formats; do
     case "$format" in
-        tar)
-            tar -cf test-$format.ova test-v2v-i-ova-formats.ovf disk1.vmdk
disk1.mf
-            ;;
         zip)
             zip -r test test-v2v-i-ova-formats.ovf disk1.vmdk disk1.mf
             mv test.zip test-$format.ova
diff --git a/v2v/test-v2v-i-ova-subfolders.expected2
b/v2v/test-v2v-i-ova-subfolders.expected2
new file mode 100644
index 000000000..87996d202
--- /dev/null
+++ b/v2v/test-v2v-i-ova-subfolders.expected2
@@ -0,0 +1,18 @@
+Source guest information (--print-source option):
+
+    source name: 2K8R2EESP1_2_Medium
+hypervisor type: vmware
+         memory: 1073741824 (bytes)
+       nr vCPUs: 1
+   CPU features: 
+       firmware: uefi
+        display: 
+          video: 
+          sound: 
+disks:
+	json:{ "file": { "driver": "raw",
"offset": 2048, "size": 10240, "file": {
"filename": "test.ova" } } } (vmdk) [scsi]
+removable media:
+	CD-ROM [ide] in slot 0
+NICs:
+	Network "Network adapter 1"
+
diff --git a/v2v/test-v2v-i-ova-subfolders.sh b/v2v/test-v2v-i-ova-subfolders.sh
index 4fd1acea3..422959036 100755
--- a/v2v/test-v2v-i-ova-subfolders.sh
+++ b/v2v/test-v2v-i-ova-subfolders.sh
@@ -56,10 +56,17 @@ popd
 # normalize the output.
 $VG virt-v2v --debug-gc --quiet \
     -i ova $d/test.ova \
-    --print-source |
-sed 's,[^ \t]*\(subfolder/disk.*\.vmdk\),\1,' > $d/source
+    --print-source > $d/source
 
 # Check the parsed source is what we expect.
-diff -u test-v2v-i-ova-subfolders.expected $d/source
+if qemu_is_version 2 8 ; then
+    # normalize the output
+    sed -i -e "s,\"$d/,\"," $d/source
+    diff -u test-v2v-i-ova-subfolders.expected2 $d/source
+else
+    # normalize the output
+    sed -i -e 's,[^ \t]*\(subfolder/disk.*\.vmdk\),\1,' $d/source
+    diff -u test-v2v-i-ova-subfolders.expected $d/source
+fi
 
 rm -rf $d
diff --git a/v2v/test-v2v-i-ova-tar.expected b/v2v/test-v2v-i-ova-tar.expected
new file mode 100644
index 000000000..7049aeecc
--- /dev/null
+++ b/v2v/test-v2v-i-ova-tar.expected
@@ -0,0 +1,18 @@
+Source guest information (--print-source option):
+
+    source name: 2K8R2EESP1_2_Medium
+hypervisor type: vmware
+         memory: 1073741824 (bytes)
+       nr vCPUs: 1
+   CPU features: 
+       firmware: uefi
+        display: 
+          video: 
+          sound: 
+disks:
+	disk1.vmdk (vmdk) [scsi]
+removable media:
+	CD-ROM [ide] in slot 0
+NICs:
+	Network "Network adapter 1"
+
diff --git a/v2v/test-v2v-i-ova-tar.expected2 b/v2v/test-v2v-i-ova-tar.expected2
new file mode 100644
index 000000000..1aa54dc37
--- /dev/null
+++ b/v2v/test-v2v-i-ova-tar.expected2
@@ -0,0 +1,18 @@
+Source guest information (--print-source option):
+
+    source name: 2K8R2EESP1_2_Medium
+hypervisor type: vmware
+         memory: 1073741824 (bytes)
+       nr vCPUs: 1
+   CPU features: 
+       firmware: uefi
+        display: 
+          video: 
+          sound: 
+disks:
+	json:{ "file": { "driver": "raw",
"offset": 9216, "size": 10240, "file": {
"filename": "test-tar.ova" } } } (vmdk) [scsi]
+removable media:
+	CD-ROM [ide] in slot 0
+NICs:
+	Network "Network adapter 1"
+
diff --git a/v2v/test-v2v-i-ova-tar.ovf b/v2v/test-v2v-i-ova-tar.ovf
new file mode 100644
index 000000000..4827c7e9b
--- /dev/null
+++ b/v2v/test-v2v-i-ova-tar.ovf
@@ -0,0 +1,138 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Envelope vmw:buildId="build-1750787"
xmlns="http://schemas.dmtf.org/ovf/envelope/1"
xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common"
xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1"
xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData"
xmlns:vmw="http://www.vmware.com/schema/ovf"
xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  <References>
+    <File ovf:href="disk1.vmdk" ovf:id="file1"
ovf:size="7804077568"/>
+  </References>
+  <DiskSection>
+    <Info>Virtual disk information</Info>
+    <Disk ovf:capacity="50" ovf:capacityAllocationUnits="byte
* 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"
ovf:populatedSize="18975752192"/>
+  </DiskSection>
+  <NetworkSection>
+    <Info>The list of logical networks</Info>
+    <Network ovf:name="PG-VLAN60">
+      <Description>The PG-VLAN60 network</Description>
+    </Network>
+  </NetworkSection>
+  <VirtualSystem ovf:id="2K8R2EESP1_2_Medium">
+    <Info>A virtual machine</Info>
+    <Name>2K8R2EESP1_2_Medium</Name>
+    <OperatingSystemSection ovf:id="103"
vmw:osType="windows7Server64Guest">
+      <Info>The kind of installed guest operating system</Info>
+      <Description>Microsoft Windows Server 2008 R2
(64-bit)</Description>
+    </OperatingSystemSection>
+    <VirtualHardwareSection>
+      <Info>Virtual hardware requirements</Info>
+      <System>
+        <vssd:ElementName>Virtual Hardware
Family</vssd:ElementName>
+        <vssd:InstanceID>0</vssd:InstanceID>
+       
<vssd:VirtualSystemIdentifier>2K8R2EESP1_2_Medium</vssd:VirtualSystemIdentifier>
+        <vssd:VirtualSystemType>vmx-10</vssd:VirtualSystemType>
+      </System>
+      <Item>
+        <rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
+        <rasd:Description>Number of Virtual CPUs</rasd:Description>
+        <rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
+        <rasd:InstanceID>1</rasd:InstanceID>
+        <rasd:ResourceType>3</rasd:ResourceType>
+        <rasd:VirtualQuantity>1</rasd:VirtualQuantity>
+      </Item>
+      <Item>
+        <rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
+        <rasd:Description>Memory Size</rasd:Description>
+        <rasd:ElementName>1024MB of memory</rasd:ElementName>
+        <rasd:InstanceID>2</rasd:InstanceID>
+        <rasd:ResourceType>4</rasd:ResourceType>
+        <rasd:VirtualQuantity>1024</rasd:VirtualQuantity>
+      </Item>
+      <Item>
+        <rasd:Address>0</rasd:Address>
+        <rasd:Description>SCSI Controller</rasd:Description>
+        <rasd:ElementName>SCSI controller 0</rasd:ElementName>
+        <rasd:InstanceID>3</rasd:InstanceID>
+        <rasd:ResourceSubType>lsilogicsas</rasd:ResourceSubType>
+        <rasd:ResourceType>6</rasd:ResourceType>
+        <vmw:Config ovf:required="false"
vmw:key="slotInfo.pciSlotNumber" vmw:value="160"/>
+      </Item>
+      <Item>
+        <rasd:Address>1</rasd:Address>
+        <rasd:Description>IDE Controller</rasd:Description>
+        <rasd:ElementName>IDE 1</rasd:ElementName>
+        <rasd:InstanceID>4</rasd:InstanceID>
+        <rasd:ResourceType>5</rasd:ResourceType>
+      </Item>
+      <Item>
+        <rasd:Address>0</rasd:Address>
+        <rasd:Description>IDE Controller</rasd:Description>
+        <rasd:ElementName>IDE 0</rasd:ElementName>
+        <rasd:InstanceID>5</rasd:InstanceID>
+        <rasd:ResourceType>5</rasd:ResourceType>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>Video card</rasd:ElementName>
+        <rasd:InstanceID>6</rasd:InstanceID>
+        <rasd:ResourceType>24</rasd:ResourceType>
+        <vmw:Config ovf:required="false"
vmw:key="enable3DSupport" vmw:value="false"/>
+        <vmw:Config ovf:required="false"
vmw:key="use3dRenderer" vmw:value="automatic"/>
+        <vmw:Config ovf:required="false"
vmw:key="useAutoDetect" vmw:value="true"/>
+        <vmw:Config ovf:required="false"
vmw:key="videoRamSizeInKB" vmw:value="4096"/>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>VMCI device</rasd:ElementName>
+        <rasd:InstanceID>7</rasd:InstanceID>
+        <rasd:ResourceSubType>vmware.vmci</rasd:ResourceSubType>
+        <rasd:ResourceType>1</rasd:ResourceType>
+        <vmw:Config ovf:required="false"
vmw:key="allowUnrestrictedCommunication"
vmw:value="false"/>
+        <vmw:Config ovf:required="false"
vmw:key="slotInfo.pciSlotNumber" vmw:value="32"/>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AddressOnParent>0</rasd:AddressOnParent>
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>CD/DVD drive 1</rasd:ElementName>
+        <rasd:InstanceID>8</rasd:InstanceID>
+        <rasd:Parent>4</rasd:Parent>
+       
<rasd:ResourceSubType>vmware.cdrom.atapi</rasd:ResourceSubType>
+        <rasd:ResourceType>15</rasd:ResourceType>
+      </Item>
+      <Item>
+        <rasd:AddressOnParent>0</rasd:AddressOnParent>
+        <rasd:ElementName>Hard disk 1</rasd:ElementName>
+        <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
+        <rasd:InstanceID>9</rasd:InstanceID>
+        <rasd:Parent>3</rasd:Parent>
+        <rasd:ResourceType>17</rasd:ResourceType>
+        <vmw:Config ovf:required="false"
vmw:key="backing.writeThrough" vmw:value="false"/>
+      </Item>
+      <Item>
+        <rasd:AddressOnParent>7</rasd:AddressOnParent>
+        <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
+        <rasd:Connection>PG-VLAN60</rasd:Connection>
+        <rasd:Description>E1000 ethernet adapter on
&quot;PG-VLAN60&quot;</rasd:Description>
+        <rasd:ElementName>Network adapter 1</rasd:ElementName>
+        <rasd:InstanceID>11</rasd:InstanceID>
+        <rasd:ResourceSubType>E1000</rasd:ResourceSubType>
+        <rasd:ResourceType>10</rasd:ResourceType>
+        <vmw:Config ovf:required="false"
vmw:key="slotInfo.pciSlotNumber" vmw:value="33"/>
+        <vmw:Config ovf:required="false"
vmw:key="wakeOnLanEnabled" vmw:value="true"/>
+      </Item>
+      <vmw:Config ovf:required="false"
vmw:key="cpuHotAddEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="cpuHotRemoveEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="firmware" vmw:value="efi"/>
+      <vmw:Config ovf:required="false"
vmw:key="virtualICH7MPresent" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="virtualSMCPresent" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="memoryHotAddEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="nestedHVEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="powerOpInfo.powerOffType" vmw:value="soft"/>
+      <vmw:Config ovf:required="false"
vmw:key="powerOpInfo.resetType" vmw:value="soft"/>
+      <vmw:Config ovf:required="false"
vmw:key="powerOpInfo.standbyAction"
vmw:value="checkpoint"/>
+      <vmw:Config ovf:required="false"
vmw:key="powerOpInfo.suspendType" vmw:value="hard"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.afterPowerOn" vmw:value="true"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.afterResume" vmw:value="true"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.beforeGuestShutdown" vmw:value="true"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.beforeGuestStandby" vmw:value="true"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.syncTimeWithHost" vmw:value="false"/>
+      <vmw:Config ovf:required="false"
vmw:key="tools.toolsUpgradePolicy"
vmw:value="upgradeAtPowerCycle"/>
+    </VirtualHardwareSection>
+  </VirtualSystem>
+</Envelope>                                 
diff --git a/v2v/test-v2v-i-ova-tar.sh b/v2v/test-v2v-i-ova-tar.sh
new file mode 100755
index 000000000..c3b0588f4
--- /dev/null
+++ b/v2v/test-v2v-i-ova-tar.sh
@@ -0,0 +1,72 @@
+#!/bin/bash -
+# libguestfs virt-v2v test script
+# Copyright (C) 2014-2016 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# Test -i ova option with ova file compressed in different ways
+
+unset CDPATH
+export LANG=C
+set -e
+
+if [ -n "$SKIP_TEST_V2V_I_OVA_FORMATS_SH" ]; then
+    echo "$0: test skipped because environment variable is set"
+    exit 77
+fi
+
+if [ "$(guestfish get-backend)" = "uml" ]; then
+    echo "$0: test skipped because UML backend does not support
network"
+    exit 77
+fi
+
+export VIRT_TOOLS_DATA_DIR="$srcdir/../test-data/fake-virt-tools"
+
+. $srcdir/../test-data/test-utils.sh
+
+d=test-v2v-i-ova-tar.d
+rm -rf $d
+mkdir $d
+
+pushd $d
+
+# Create a phony OVA.  This is only a test of source parsing, not
+# conversion, so the contents of the disks doesn't matter.
+truncate -s 10k disk1.vmdk
+sha=`do_sha1 disk1.vmdk`
+echo -e "SHA1(disk1.vmdk)= $sha\r" > disk1.mf
+cp ../test-v2v-i-ova-tar.ovf .
+tar -cf test-tar.ova test-v2v-i-ova-tar.ovf disk1.vmdk disk1.mf
+
+popd
+
+# Run virt-v2v but only as far as the --print-source stage
+$VG virt-v2v --debug-gc --quiet \
+    -i ova $d/test-tar.ova \
+    --print-source > $d/source
+
+# Check the parsed source is what we expect.
+if qemu_is_version 2 8 ; then
+    # normalize the output
+    sed -i -e "s,\"$d/,\"," $d/source
+    diff -u test-v2v-i-ova-tar.expected2 $d/source
+else
+    # normalize the output
+    sed -i -e 's,[^ \t]*\(disk.*.vmdk\),\1,' $d/source
+    diff -u test-v2v-i-ova-tar.expected $d/source
+fi
+
+
+rm -rf $d
diff --git a/v2v/test-v2v-i-ova-two-disks.expected2
b/v2v/test-v2v-i-ova-two-disks.expected2
new file mode 100644
index 000000000..b12ca1bd6
--- /dev/null
+++ b/v2v/test-v2v-i-ova-two-disks.expected2
@@ -0,0 +1,19 @@
+Source guest information (--print-source option):
+
+    source name: 2K8R2EESP1_2_Medium
+hypervisor type: vmware
+         memory: 1073741824 (bytes)
+       nr vCPUs: 1
+   CPU features: 
+       firmware: bios
+        display: 
+          video: 
+          sound: 
+disks:
+	json:{ "file": { "driver": "raw",
"offset": 9728, "size": 10240, "file": {
"filename": "test.ova" } } } (vmdk) [scsi]
+	json:{ "file": { "driver": "raw",
"offset": 21504, "size": 102400, "file": {
"filename": "test.ova" } } } (vmdk) [scsi]
+removable media:
+	CD-ROM [ide] in slot 0
+NICs:
+	Network "Network adapter 1"
+
diff --git a/v2v/test-v2v-i-ova-two-disks.sh b/v2v/test-v2v-i-ova-two-disks.sh
index 26dd19860..310aff1b9 100755
--- a/v2v/test-v2v-i-ova-two-disks.sh
+++ b/v2v/test-v2v-i-ova-two-disks.sh
@@ -60,10 +60,17 @@ popd
 # normalize the output.
 $VG virt-v2v --debug-gc --quiet \
     -i ova $d/test.ova \
-    --print-source |
-sed 's,[^ \t]*\(disk.*.vmdk\),\1,' > $d/source
+    --print-source  > $d/source
 
 # Check the parsed source is what we expect.
-diff -u test-v2v-i-ova-two-disks.expected $d/source
+if qemu_is_version 2 8 ; then
+    # normalize the output
+    sed -i -e "s,\"$d/,\"," $d/source
+    diff -u test-v2v-i-ova-two-disks.expected2 $d/source
+else
+    # normalize the output
+    sed -i -e 's,[^ \t]*\(disk.*.vmdk\),\1,' $d/source
+    diff -u test-v2v-i-ova-two-disks.expected $d/source
+fi
 
 rm -rf $d
-- 
2.11.0

I will push these in an hour or two after the slow tests have run.

Note that I have made a few (only stylistic) changes to your commits,
so if you `git pull --rebase' then you will see conflicts.

Thanks for this important contribution to virt-v2v.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Mon, Feb 06, 2017 at 11:06:46AM +0300, Roman Kagan
wrote:
> On Sat, Feb 04, 2017 at 03:10:44PM +0100, Tomáš Golembiovský wrote:
> > The function returns version of qemu-img as a tuple (major, minor), or
> > the value (0,9) in case there was an error detecting the version.
> 
> Depending on QEMU version is extremly unreliable.  Vendors often stick
> with fairly old versions but backport important features (you at RedHat
> know that, don't you ;)  So it's not unlikely that e.g. a number of
> future RHEV/RHEL releases won't use this new feature even though the
> included QEMU already supports it.
> 
> IMO a simpler and more reliable check is to construct a command line for
> qemu-img that is lightweight to execute and succeeds if the option is
> supported or fails otherwise.
> 
> [ sorry for being so late with this ]
Yes, I agree.  My quick testing shows that the following command
should be sufficent to test for the "offset" parameter:

$ truncate -s1024 /tmp/test2
$ qemu-img create /tmp/test -f qcow2 -b
'json:{"offset":512,"file":{"filename":"/tmp/test2"}}'

On RHEL 7.3 it fails with:

qemu-img: /tmp/test: Block format 'raw' used by device ''
doesn't support the option 'offset'

and return code 1.

(Obviously we'd have to use real temporary files)

I'll see if I can make a patch ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Mon, 6 Feb 2017 15:25:53 +0000
"Richard W.M. Jones" <rjones@redhat.com> wrote:
> On Mon, Feb 06, 2017 at 11:06:46AM +0300, Roman Kagan wrote:
> > On Sat, Feb 04, 2017 at 03:10:44PM +0100, Tomáš Golembiovský wrote:  
> > > The function returns version of qemu-img as a tuple (major,
minor), or
> > > the value (0,9) in case there was an error detecting the version.
> > 
> > Depending on QEMU version is extremly unreliable.  Vendors often stick
> > with fairly old versions but backport important features (you at
RedHat
> > know that, don't you ;)  So it's not unlikely that e.g. a
number of
> > future RHEV/RHEL releases won't use this new feature even though
the
> > included QEMU already supports it.
> > 
> > IMO a simpler and more reliable check is to construct a command line
for
> > qemu-img that is lightweight to execute and succeeds if the option is
> > supported or fails otherwise.
> > 
> > [ sorry for being so late with this ]  
> 
Good point. I didn't think this through.
> Yes, I agree.  My quick testing shows that the following command
> should be sufficent to test for the "offset" parameter:
> 
> $ truncate -s1024 /tmp/test2
> $ qemu-img create /tmp/test -f qcow2 -b
'json:{"offset":512,"file":{"filename":"/tmp/test2"}}'
> 
> On RHEL 7.3 it fails with:
> 
> qemu-img: /tmp/test: Block format 'raw' used by device ''
doesn't support the option 'offset'
> 
> and return code 1.
> 
> (Obviously we'd have to use real temporary files)
No we don't. Something like this also works:

    qemu-img info 'json:{ "offset": 0, "file": {
"filename": "/dev/null" } }'


    Tomas

> I'll see if I can make a patch ...
> 
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org

-- 
Tomáš Golembiovský <tgolembi@redhat.com>

On Sat, Feb 04, 2017 at 03:10:46PM +0100, Tomáš Golembiovský
wrote:
> We don't have to always extract all files from the OVA archive. The
OVA,
> as defined in the standard, is plain tar. We can work directly over the
> tar archive if we use correct 'offset' and 'size' options
when defining
> the backing file for QEMU. This puts much lower requirement on available
> disk space.
> 
> Since the virt-v2v behaviour for OVA input now depends on QEMU version
> available this affects some of the tests. Expected result of the
> affected also has to depend on the QEMU used thus such tests will have
> two *.expected files.
> 
> Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com>
This commit (I think) breaks virt-v2v when run with the libvirt
backend using an OVA input file.  The error from v2v/test-v2v-i-ova.sh
is:

  Original error from libvirt: internal error: missing parser implementation 
  for JSON backing volume driver 'raw' [code=1 int1=-1]

which I think is caused because libvirt doesn't understand a backing
file with the format 'json:{"driver":"raw",...}'. 
Full log is
attached.  This is similar but not exactly the same as this bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1134878

I have added a workaround in the test for now, but please look at
whether this can be fixed more permanently.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

--a8Wt8u1KmwUX3Y2C
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="test-v2v-i-ova.sh.log"
Content-Transfer-Encoding: quoted-printable

libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x55c0a01e50a0, program =3D
guestfish
libguestfs: trace: get_backend
libguestfs: trace: get_backend =3D "libvirt"
libguestfs: trace: shutdown
libguestfs: trace: shutdown =3D 0
libguestfs: trace: close
libguestfs: closing guestfs handle 0x55c0a01e50a0 (state 0)
~/d/libguestfs/v2v ~/d/libguestfs/v2v
~/d/libguestfs/v2v
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x10c4c60, program =3D virt-v2v
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir =3D "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x10c50b0, program =3D virt-v2v
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir =3D "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x10c5660, program =3D virt-v2v
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir =3D "/home/rjones/d/libguestfs/tmp"
[   0.0] Opening the source -i ova test-v2v-i-ova.d/test-ova.ova
[   0.1] Creating an overlay to protect the source from being modified
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x11406d0, program =3D virt-v2v
libguestfs: trace: disk_has_backing_file
"/home/rjones/d/libguestfs/tmp/v2vovlb76359.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/3
libguestfs: parse_json: qemu-img info JSON output:\n{\n   
"backing-filename-format": "vmdk",\n   
"virtual-size": 536870912,\n    "filename":
"/dev/fd/3",\n    "cluster-size": 65536,\n   
"format": "qcow2",\n    "actual-size": 200704,\n  
"format-specific": {\n        "type": "qcow2",\n  
"data": {\n            "compat": "1.1",\n         
"lazy-refcounts": false,\n            "refcount-bits": 16,\n
"corrupt": false\n        }\n    },\n   
"full-backing-filename": "json:{ \"file\": {
\"driver\": \"raw\", \"offset\": 10752,
\"size\": 4063232, \"file\": { \"filename\":
\"test-v2v-i-ova.d/test-ova.ova\" } } }",\n   
"backing-filename": "json:{ \"file\": {
\"driver\": \"raw\", \"offset\": 10752,
\"size\": 4063232, \"file\": { \"filename\":
\"test-v2v-i-ova.d/test-ova.ova\" } } }",\n   
"dirty-flag": false\n}\n\n
libguestfs: trace: disk_has_backing_file =3D 1
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x113fde0, program =3D virt-v2v
libguestfs: trace: disk_virtual_size
"/home/rjones/d/libguestfs/tmp/v2vovlb76359.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/3
libguestfs: parse_json: qemu-img info JSON output:\n{\n   
"backing-filename-format": "vmdk",\n   
"virtual-size": 536870912,\n    "filename":
"/dev/fd/3",\n    "cluster-size": 65536,\n   
"format": "qcow2",\n    "actual-size": 200704,\n  
"format-specific": {\n        "type": "qcow2",\n  
"data": {\n            "compat": "1.1",\n         
"lazy-refcounts": false,\n            "refcount-bits": 16,\n
"corrupt": false\n        }\n    },\n   
"full-backing-filename": "json:{ \"file\": {
\"driver\": \"raw\", \"offset\": 10752,
\"size\": 4063232, \"file\": { \"filename\":
\"test-v2v-i-ova.d/test-ova.ova\" } } }",\n   
"backing-filename": "json:{ \"file\": {
\"driver\": \"raw\", \"offset\": 10752,
\"size\": 4063232, \"file\": { \"filename\":
\"test-v2v-i-ova.d/test-ova.ova\" } } }",\n   
"dirty-flag": false\n}\n\n
libguestfs: trace: disk_virtual_size =3D 536870912
[   0.1] Initializing the target -o local -os test-v2v-i-ova.d
[   0.1] Opening the overlay
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose =3D 0
libguestfs: trace: set_tmpdir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_tmpdir =3D 0
libguestfs: trace: set_cachedir "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: set_cachedir =3D 0
libguestfs: trace: set_path "/home/rjones/d/libguestfs/appliance"
libguestfs: trace: set_path =3D 0
libguestfs: create: flags =3D 0, handle =3D 0x1140130, program =3D virt-v2v
libguestfs: trace: set_identifier "v2v"
libguestfs: trace: v2v: set_identifier =3D 0
libguestfs: trace: v2v: get_memsize
libguestfs: trace: v2v: get_memsize =3D 500
libguestfs: trace: v2v: set_memsize 2000
libguestfs: trace: v2v: set_memsize =3D 0
libguestfs: trace: v2v: set_network true
libguestfs: trace: v2v: set_network =3D 0
libguestfs: trace: v2v: add_drive
"/home/rjones/d/libguestfs/tmp/v2vovlb76359.qcow2"
"format:qcow2" "cachemode:unsafe"
"discard:besteffort" "copyonread:true"
libguestfs: trace: v2v: add_drive =3D 0
libguestfs: trace: v2v: launch
libguestfs: trace: v2v: get_tmpdir
libguestfs: trace: v2v: get_tmpdir =3D "/home/rjones/d/libguestfs/tmp"
libguestfs: trace: v2v: version
libguestfs: trace: v2v: version =3D <struct guestfs_version =3D major: 1,
minor: 35, release: 21, extra: , >
libguestfs: trace: v2v: get_backend
libguestfs: trace: v2v: get_backend =3D "libvirt"
libguestfs: launch: program=3Dvirt-v2v
libguestfs: launch: identifier=3Dv2v
libguestfs: launch: version=3D1.35.21
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=3Dlibvirt
libguestfs: launch: tmpdir=3D/home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y
libguestfs: launch: umask=3D0002
libguestfs: launch: euid=3D1000
libguestfs: libvirt version =3D 3000000 (3.0.0)
libguestfs: guest random name =3D guestfs-rkoxuu2n9eeycio5
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI =3D qemu:///session, auth =3D
default+wrapper, flags =3D 0
libguestfs: successfully opened libvirt handle: conn =3D 0x1177320
libguestfs: qemu version (reported by libvirt) =3D 2008000 (2.8.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: trace: v2v: get_backend_setting "force_tcg"
libguestfs: trace: v2v: get_backend_setting =3D NULL (error)
libguestfs: trace: v2v: get_backend_setting "internal_libvirt_label"
libguestfs: trace: v2v: get_backend_setting =3D NULL (error)
libguestfs: trace: v2v: get_backend_setting
"internal_libvirt_imagelabel"
libguestfs: trace: v2v: get_backend_setting =3D NULL (error)
libguestfs: trace: v2v: get_backend_setting
"internal_libvirt_norelabel_disks"
libguestfs: trace: v2v: get_backend_setting =3D NULL (error)
libguestfs: trace: v2v: get_backend_setting "network_bridge"
libguestfs: trace: v2v: get_backend_setting =3D NULL (error)
libguestfs: build appliance
libguestfs: trace: v2v: get_cachedir
libguestfs: trace: v2v: get_cachedir =3D
"/home/rjones/d/libguestfs/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock
/home/rjones/d/libguestfs/tmp/.guestfs-1000/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /home/rjones/d/libguestfs/appliance/supermin.d
libguestfs: command: run: \ -o
/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d
supermin: version: 5.1.17
supermin: rpm: detected RPM version 4.13
supermin: package handler: fedora/rpm
supermin: acquiring lock on /home/rjones/d/libguestfs/tmp/.guestfs-1000/lock
supermin: if-newer: output does not need rebuilding
libguestfs: finished building supermin appliance
libguestfs: trace: v2v: disk_create
"/home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y/overlay1"
"qcow2" -1
"backingfile:/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/root"
"backingformat:raw"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o
backing_file=3D/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/root,backing_fmt=3Draw
libguestfs: command: run: \
/home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y/overlay1
Formatting '/home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y/overlay1',
fmt=3Dqcow2 size=3D4294967296
backing_file=3D/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/root
backing_fmt=3Draw encryption=3Doff cluster_size=3D65536 lazy_refcounts=3Doff
refcount_bits=3D16
libguestfs: trace: v2v: disk_create =3D 0
libguestfs: trace: v2v: get_sockdir
libguestfs: trace: v2v: get_sockdir =3D "/run/user/1000"
libguestfs: create libvirt XML
libguestfs: trace: v2v: get_cachedir
libguestfs: trace: v2v: get_cachedir =3D
"/home/rjones/d/libguestfs/tmp"
libguestfs: libvirt XML:\n<?xml version=3D"1.0"?>\n<domain
type=3D"kvm"
xmlns:qemu=3D"http://libvirt.org/schemas/domain/qemu/1.0">\n 
<name>guestfs-rkoxuu2n9eeycio5</name>\n  <memory
unit=3D"MiB">2000</memory>\n  <currentMemory
unit=3D"MiB">2000</currentMemory>\n  <cpu
mode=3D"host-passthrough">\n    <model
fallback=3D"allow"/>\n  </cpu>\n 
<vcpu>1</vcpu>\n  <clock offset=3D"utc">\n   
<timer name=3D"rtc" tickpolicy=3D"catchup"/>\n   
<timer name=3D"pit" tickpolicy=3D"delay"/>\n   
<timer name=3D"hpet" present=3D"no"/>\n 
</clock>\n  <os>\n    <type>hvm</type>\n   
<kernel>/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel</kernel>\n
<initrd>/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/initrd</initrd>\n
<cmdline>panic=3D1 console=3DttyS0 edd=3Doff udevtimeout=3D6000
udev.event-timeout=3D6000 no_timer_check printk.time=3D1 cgroup_disable=3Dmemory
usbcore.nousb cryptomgr.notests tsc=3Dreliable 8250.nr_uarts=3D1 root=3D/dev/sdb
selinux=3D0 guestfs_verbose=3D1 guestfs_network=3D1 TERM=3Dxterm-256color
guestfs_identifier=3Dv2v</cmdline>\n    <bios
useserial=3D"yes"/>\n  </os>\n 
<on_reboot>destroy</on_reboot>\n  <devices>\n    <rng
model=3D"virtio">\n      <backend
model=3D"random">/dev/urandom</backend>\n    </rng>\n  
<controller type=3D"scsi" index=3D"0"
model=3D"virtio-scsi"/>\n    <disk device=3D"disk"
type=3D"file">\n      <source
file=3D"/home/rjones/d/libguestfs/tmp/v2vovlb76359.qcow2"/>\n     
<target dev=3D"sda" bus=3D"scsi"/>\n      <driver
name=3D"qemu" type=3D"qcow2" cache=3D"unsafe"
discard=3D"unmap"/>\n      <address type=3D"drive"
controller=3D"0" bus=3D"0" target=3D"0"
unit=3D"0"/>\n    </disk>\n    <disk
type=3D"file" device=3D"disk">\n      <source
file=3D"/home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y/overlay1"/>\n
<target dev=3D"sdb" bus=3D"scsi"/>\n      <driver
name=3D"qemu" type=3D"qcow2"
cache=3D"unsafe"/>\n      <address type=3D"drive"
controller=3D"0" bus=3D"0" target=3D"1"
unit=3D"0"/>\n      <shareable/>\n    </disk>\n   
<serial type=3D"unix">\n      <source
mode=3D"connect"
path=3D"/run/user/1000/libguestfsnN9AkE/console.sock"/>\n     
<target port=3D"0"/>\n    </serial>\n    <channel
type=3D"unix">\n      <source mode=3D"connect"
path=3D"/run/user/1000/libguestfsnN9AkE/guestfsd.sock"/>\n     
<target type=3D"virtio"
name=3D"org.libguestfs.channel.0"/>\n    </channel>\n   
<interface type=3D"bridge">\n      <source
bridge=3D"virbr0"/>\n      <model
type=3D"virtio"/>\n    </interface>\n    <controller
type=3D"usb" model=3D"none"/>\n    <memballoon
model=3D"none"/>\n  </devices>\n  <qemu:commandline>\n 
<qemu:env name=3D"TMPDIR"
value=3D"/home/rjones/d/libguestfs/tmp"/>\n 
</qemu:commandline>\n</domain>\n
libguestfs: trace: v2v: get_cachedir
libguestfs: trace: v2v: get_cachedir =3D
"/home/rjones/d/libguestfs/tmp"
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /home/rjones/d/libguestfs/tmp/.guestfs-1000
libguestfs: /home/rjones/d/libguestfs/tmp/.guestfs-1000:
libguestfs: total 92
libguestfs: drwxr-xr-x. 3 rjones rjones unconfined_u:object_r:user_tmp_t:s0 
4096 Feb  7 21:02 .
libguestfs: drwxrwxr-x. 6 rjones rjones system_u:object_r:tmp_t:s0         
45056 Feb  7 21:02 ..
libguestfs: drwxr-xr-x. 2 rjones rjones unconfined_u:object_r:user_tmp_t:s0 
4096 Feb  7 20:58 appliance.d
libguestfs: -rw-r--r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0    
0 Feb  7 17:41 lock
libguestfs: -rw-rw-r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0
10027 Feb  7 18:26 qemu.devices
libguestfs: -rw-rw-r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0
23588 Feb  7 18:26 qemu.help
libguestfs: -rw-rw-r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0   
35 Feb  7 18:26 qemu.stat
libguestfs:=20
libguestfs: /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d:
libguestfs: total 422532
libguestfs: drwxr-xr-x. 2 rjones rjones unconfined_u:object_r:user_tmp_t:s0     
4096 Feb  7 20:58 .
libguestfs: drwxr-xr-x. 3 rjones rjones unconfined_u:object_r:user_tmp_t:s0     
4096 Feb  7 21:02 ..
libguestfs: -rw-r--r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0    
521728 Feb  7 21:02 initrd
libguestfs: -rwxr-xr-x. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0   
7172184 Feb  7 21:02 kernel
libguestfs: -rw-r--r--. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0
4294967296 Feb  7 21:02 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /run/user/1000/libguestfsnN9AkE
libguestfs: total 0
libguestfs: drwx------. 2 rjones rjones unconfined_u:object_r:user_tmp_t:s0  80
Feb  7 21:02 .
libguestfs: drwx------. 7 rjones rjones system_u:object_r:user_tmp_t:s0     160
Feb  7 21:02 ..
libguestfs: srwxrwxr-x. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0   0
Feb  7 21:02 console.sock
libguestfs: srwxrwxr-x. 1 rjones rjones unconfined_u:object_r:user_tmp_t:s0   0
Feb  7 21:02 guestfsd.sock
libguestfs: launch libvirt guest
libguestfs: trace: v2v: launch =3D -1 (error)
virt-v2v: error: libguestfs error: could not create appliance through=20
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=3Ddirect

Original error from libvirt: internal error: missing parser implementation=20
for JSON backing volume driver 'raw' [code=3D1 int1=3D-1]

If reporting bugs, run virt-v2v with debugging enabled and include the=20
complete output:

  virt-v2v -v -x [...]
libguestfs: trace: close
libguestfs: closing guestfs handle 0x10c4c60 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x10c50b0 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x10c5660 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x11406d0 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x113fde0 (state 0)
libguestfs: trace: v2v: close
libguestfs: closing guestfs handle 0x1140130 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /home/rjones/d/libguestfs/tmp/libguestfsPT4q2Y
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /run/user/1000/libguestfsnN9AkE
FAIL test-v2v-i-ova.sh (exit status: 1)

--a8Wt8u1KmwUX3Y2C--