Libguestfs - Feb 2019 - IO rate of tar-in, what can we expect on a qcow2 image?

Hi all,

We have done several speed tests on a qcow2 Linux Image to test how fast tar-in
with a big
tarball can be. Virtio seems to be active, and we get transfers in a range from
100-160MB/sec,
independent of the disk speed on the host. 

For example we had a 20 core host system with 900MB for serial writing and 350MB
for mixed
read/write on the native filesystem. We've expected a faster tar-in for
qcow2 there than on a
small test system with some slow disks and only two cores. But the rates where
nearly the same.
We tried really hard to do some optimizing and using big files inside the tar
(for emulating serial
writing), but we managed only to get those 160MB/sec, but not more. 

The IO-rate for serial writing inside a running VM-Image on the same qcow2 image
is much more
higher, dependent of course on the underlying disks of the host. Here we get
clear benefits of fast
storage.

Has anybody of you got more than those 160MB/sec with tar-in, and if so, how did
you do it?
Are there maybe some limitations in the XDR/RPC mechanisms of tar-in ?

Best regards:

Martin Blapp

On Tue, Feb 26, 2019 at 11:00:41AM +0100, Martin Blapp
wrote:
> Hi all,
> 
> We have done several speed tests on a qcow2 Linux Image to test how fast
tar-in with a big
> tarball can be. Virtio seems to be active, and we get transfers in a range
from 100-160MB/sec,
> independent of the disk speed on the host. 
> 
> For example we had a 20 core host system with 900MB for serial writing and
350MB for mixed
> read/write on the native filesystem. We've expected a faster tar-in for
qcow2 there than on a
> small test system with some slow disks and only two cores. But the rates
where nearly the same.
> We tried really hard to do some optimizing and using big files inside the
tar (for emulating serial
> writing), but we managed only to get those 160MB/sec, but not more. 
> 
> The IO-rate for serial writing inside a running VM-Image on the same qcow2
image is much more
> higher, dependent of course on the underlying disks of the host. Here we
get clear benefits of fast
> storage.
> 
> Has anybody of you got more than those 160MB/sec with tar-in, and if so,
how did you do it?
> Are there maybe some limitations in the XDR/RPC mechanisms of tar-in ?
I doubt that XDR (note: RPC is not used) is a problem.  There is a
special streaming mode used for tar-in and similar "upload" commands,
and apart from probably an extra copy or two there's very little
overhead.

Did you try changing the cache mode?

https://rwmj.wordpress.com/2013/09/02/new-in-libguestfs-allow-cache-mode-to-be-selected/

You could also try running virt-rescue to get direct access to an
appliance and run some tests from within that environment.  This gives
you an idea of how qemu+virtio is performing directly, without any
possible libguestfs overhead.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/