Libguestfs - Nov 2019 - [PATCH] v2v: Optimize convert for images with small holes

"qemu-img convert" detects zeroes in allocated areas and punch holes
in
the destination image. This may save space on the destination image, but
slows down conversion when using outputs such as rhv-upload, which have
very large overhead per requests.

Using the -S flag, we can treat small areas filled with zeroes as data,
limiting the number of requests, and speeding the operation.

Here is an example converting Fedora 30 image:

$ virt-builder fedora-30 -o src.img
...

$ qemu-img map -f raw --output json src.img | wc -l
213

$ qemu-img convert -f raw -O raw -t none -T none src.img dst.img

$ qemu-img map -f raw --output json dst.img | wc -l
1443

$ ls -lhs *.img
1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov ?1 21:48 dst.img
1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov ?1 21:46 src.img

Qemu did 1443 writes instead of 213 (5.8X). Lets repeat this conversion
with the -S option:

$ qemu-img convert -f raw -O raw -t none -T none -S 64k src.img dst.img

$ qemu-img map -f raw --output json dst.img | wc -l
213

$ ls -lhs *.img
1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov ?1 21:48 dst.img
1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov ?1 21:46 src.img

Picking a good value for -S is not easy. Testing show that 64k is best
value for this test image for limiting the number of requests:

$ for size in 4k 8k 16k 32k 64k; do \
    printf "%5s: " $size; \
    qemu-img convert -f raw -O raw -t none -T none -S $size src.img dst.img; \
    qemu-img map -f raw --output json dst.img | wc -l; \
done
? ?4k: 1443
? ?8k: 731
? 16k: 521
? 32k: 387
? 64k: 213

We need more testing with oVirt to measure the performance improvement
and pick a good value. This should probably be an option, but lets start
with a minimal change.
---

Untested. I cannot build virt-v2v on Fedora 29 since libguestfs 1.41.5
is required but not available. I can also not build libguestfs from git
becuase setting up common submodule in ./autogen.sh fails.

 v2v/v2v.ml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 4655b883..03590c9e 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -741,6 +741,7 @@ and copy_targets cmdline targets input output          (if
not (quiet ()) then [ "-p" ] else []) @
         [ "-n"; "-f"; "qcow2"; "-O";
t.target_format ] @
         (if cmdline.compressed then [ "-c" ] else []) @
+        [ "-S"; "64k" ] @
         [ overlay_file; filename ] in
       let start_time = gettimeofday () in
       if run_command cmd <> 0 then
-- 
2.20.1

On Fri, Nov 1, 2019, 22:56 Nir Soffer <nirsof at gmail.com> wrote:
> "qemu-img convert" detects zeroes in allocated areas and punch
holes in
> the destination image. This may save space on the destination image, but
> slows down conversion when using outputs such as rhv-upload, which have
> very large overhead per requests.
>
> Using the -S flag, we can treat small areas filled with zeroes as data,
> limiting the number of requests, and speeding the operation.
>
> Here is an example converting Fedora 30 image:
>
> $ virt-builder fedora-30 -o src.img
> ...
>
> $ qemu-img map -f raw --output json src.img | wc -l
> 213
>
> $ qemu-img convert -f raw -O raw -t none -T none src.img dst.img
>
> $ qemu-img map -f raw --output json dst.img | wc -l
> 1443
>
> $ ls -lhs *.img
> 1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov  1 21:48 dst.img
> 1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov  1 21:46 src.img
>
> Qemu did 1443 writes instead of 213 (5.8X). Lets repeat this conversion
> with the -S option:
>
> $ qemu-img convert -f raw -O raw -t none -T none -S 64k src.img dst.img
>
> $ qemu-img map -f raw --output json dst.img | wc -l
> 213
>
> $ ls -lhs *.img
> 1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov  1 21:48 dst.img
> 1.2G -rw-r--r--. 1 nsoffer nsoffer 6.0G Nov  1 21:46 src.img
>
> Picking a good value for -S is not easy. Testing show that 64k is best
> value for this test image for limiting the number of requests:
>
> $ for size in 4k 8k 16k 32k 64k; do \
>     printf "%5s: " $size; \
>     qemu-img convert -f raw -O raw -t none -T none -S $size src.img
> dst.img; \
>     qemu-img map -f raw --output json dst.img | wc -l; \
> done
>    4k: 1443
>    8k: 731
>   16k: 521
>   32k: 387
>   64k: 213
>
> We need more testing with oVirt to measure the performance improvement
> and pick a good value. This should probably be an option, but lets start
> with a minimal change.
> ---
>
> Untested. I cannot build virt-v2v on Fedora 29 since libguestfs 1.41.5
> is required but not available. I can also not build libguestfs from git
> becuase setting up common submodule in ./autogen.sh fails.
>
>  v2v/v2v.ml | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/v2v/v2v.ml b/v2v/v2v.ml
> index 4655b883..03590c9e 100644
> --- a/v2v/v2v.ml
> +++ b/v2v/v2v.ml
> @@ -741,6 +741,7 @@ and copy_targets cmdline targets input output >     
(if not (quiet ()) then [ "-p" ] else []) @
>          [ "-n"; "-f"; "qcow2";
"-O"; t.target_format ] @
>          (if cmdline.compressed then [ "-c" ] else []) @
> +        [ "-S"; "64k" ] @
>          [ overlay_file; filename ] in
>        let start_time = gettimeofday () in
>        if run_command cmd <> 0 then
>
I finally built this by applying the patch to libguestfs 1.40.1.

I did initial testing with local output like this:

$ ./run virt-v2v -v -x -i disk /home/nsoffer/tmp/sparse-size/src.img -o
local -of raw -os /home/nsoffer/tmp/sparse-size

Testing show that this change improves throughput, but converting with
local file system is so fast that this is not interesting.

I hope to more interesting results with rhv-upload. Hopefully I can test
this next week.

---

With 1.40.1:

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw'
'/home/nsoffer/src/libguestfs/tmp/v2vovld6b8b9.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 59718.8 M bits/sec
real copying rate: 11479.2 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1238372352 (1.2G): 98.4%

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw'
'/home/nsoffer/src/libguestfs/tmp/v2vovl70b9b6.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 65311.3 M bits/sec
real copying rate: 12567.4 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1239678976 (1.2G): 98.2%

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw'
'/home/nsoffer/src/libguestfs/tmp/v2vovl1c8ada.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 68175.6 M bits/sec
real copying rate: 13118.6 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1239678976 (1.2G): 98.2%

With 1.40.1 + patch:

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw' '-S' '64k'
'/home/nsoffer/src/libguestfs/tmp/v2vovl981419.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 70717.5 M bits/sec
real copying rate: 13763.8 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1253900288 (1.2G): 95.9%

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw' '-S' '64k'
'/home/nsoffer/src/libguestfs/tmp/v2vovl10e7e3.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 73361.8 M bits/sec
real copying rate: 14278.5 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1253900288 (1.2G): 95.9%

qemu-img 'convert' '-p' '-n' '-f'
'qcow2' '-O' 'raw' '-S' '64k'
'/home/nsoffer/src/libguestfs/tmp/v2vovldf5aaf.qcow2'
'/home/nsoffer/tmp/sparse-size/src-sda'
    (100.00/100%)
du --block-size=1 '/home/nsoffer/tmp/sparse-size/src-sda' | awk
'{print $1}'
virtual copying rate: 73106.5 M bits/sec
real copying rate: 14228.8 M bits/sec
sda: estimate 2456662831 (2.3G) versus actual 1253900288 (1.2G): 95.9%
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20191102/e05089fb/attachment.htm>

On Fri, Nov 01, 2019 at 10:56:18PM +0100, Nir Soffer
wrote:
> Untested. I cannot build virt-v2v on Fedora 29 since libguestfs 1.41.5
> is required but not available. I can also not build libguestfs from git
> becuase setting up common submodule in ./autogen.sh fails.
You can try doing:

git submodule init
git submodule update

Note if there's a common/ subdirectory left over with temporary files
in it, you have to delete it first.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Fri, Nov 01, 2019 at 10:56:18PM +0100, Nir Soffer
wrote:
> diff --git a/v2v/v2v.ml b/v2v/v2v.ml
> index 4655b883..03590c9e 100644
> --- a/v2v/v2v.ml
> +++ b/v2v/v2v.ml
> @@ -741,6 +741,7 @@ and copy_targets cmdline targets input output >     
(if not (quiet ()) then [ "-p" ] else []) @
>          [ "-n"; "-f"; "qcow2";
"-O"; t.target_format ] @
>          (if cmdline.compressed then [ "-c" ] else []) @
> +        [ "-S"; "64k" ] @
I kind of wonder if we should make this something that the output mode
can set.  However I don't think it's a big deal since (a) 64K is the
same as the standard qcow2 cluster size so it won't affect qcow2
outputs and (b) it's not a particularly large size even for raw
outputs.  Therefore:

ACK

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top