Libguestfs - Jan 2022 - Virt-v2v performance benchmarking part 3

On Mon, Jan 10, 2022 at 11:21:16AM +0100, Laszlo Ersek
wrote:
> On 01/09/22 19:51, Richard W.M. Jones wrote:
> > 
> > Part 1:
> >
https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html
> > Part 2:
> >
https://listman.redhat.com/archives/libguestfs/2022-January/msg00057.html
> > 
> > This is part 3 of my performance analysis of virt-v2v over the last
> > year.  In this email I cover conversion from VMware to a local disk
> > using VDDK.  This is a more realistic test than doing local disk to
> > local disk conversions.
> > 
> > As you can see from the new chart in the attached file [LibreOffice
> > Calc format] modular virt-v2v has got a little faster over all, with
> > conversion taking slightly longer and copying being slightly faster.
> > 
> > If you expand the hidden columns (between columns F & M) you will
also
> > see clearly the new flushing behaviour of nbdcopy, where it always
> > flushes the output to disk, versus "qemu-img convert" which
used the
> > page cache (notice the Sync times in column L).  This can make old
> > virt-v2v appear to be much faster, but the appearance is not real.
> 
> So this confirms that there is no performance regression in the
"VMware
> to a local disk using VDDK" conversion case -- this is the result
we've
> been looking for, right?
Yes, but there could _appear_ to be a regression because of the
substantial extra time taken to flush data to disk (the "Sync"
column).  I've added the Sync time to the total times, but if you were
to omit that then old virt-v2v would appear to be ~ 50 seconds faster.
I think flushing / not using the page cache is important for whole
system throughput so I want to keep this behaviour.

There is very definitely a regression for local disk to local disk raw
format which I'm looking at now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

For the raw format local disk to local disk conversion, it's possible
to regain most of the performance by adding
--request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command.  The
patch below is not suitable for going upstream but it can be used for
testing:

diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 47e6e937..ece3b7d9 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri    let cmd =
ref [] in
   List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ];
   List.push_back cmd "--flush";
+  List.push_back cmd "--request-size=16777216";
   (*List.push_back cmd "--verbose";*)
   if not (quiet ()) then List.push_back cmd "--progress";
   if output_alloc = Types.Preallocated then List.push_back cmd
"--allocated";

The problem is of course this is a pessimisation for other
conversions.  It's known to make at least qcow2 to qcow2, and all VDDK
conversions worse.  So we'd have to make it conditional on doing a raw
format local conversion, which is a pretty ugly hack.  Even worse, the
exact size (16M) varies for me when I test this on different machines
and HDDs vs SSDs.  On my very fast AMD machine with an SSD, the
nbdcopy default request size (256K) is fastest and larger sizes are a
very slightly slower.

I can imagine an "adaptive nbdcopy" which adjusts these parameters
while copying in order to find the best performance.  A little bit
hard to implement ...

I'm also still wondering exactly why a larger request size is better
in this case.  You can easily reproduce the effect using the attached
test script and adjusting --request-size.  You'll need to build the
standard test guest, see part 1.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.sh
Type: application/x-sh
Size: 578 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20220110/d5851b99/attachment.sh>