libvirt users - Jan 2014 - Yet another disk I/O performance issue

Hi all,

I'm running a VM using libvirt+KVM and I have a disk performance issue.

The host is the following:
4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy,
libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2.

The guest:
1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format.

When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40" I
get
around 500 MB/s on bare metal, while only around 30 MB/s inside the VM.

I'm trying to get something more out of the virtualization layer, I hope
that there's room for improvement.

I'm using virtio, I aready set cache='none' and io='native'
in the domain
definition. Both host and guest are using deadline as I/O scheduler. The VM
uses an ext4 filesystem, while the image is saved on an ext3 disk. I
mounted the host and guest filesystems specifying nodiratime and noatime
options. Even if I convert the image to raw format nothing changes.
I didn't mess with iotune nor blockio.

Is there something that I overlooked or any other suggestion?

Thanks in advance for your help.

Matteo



-- 
A refund for defective software might be nice, except it would bankrupt the
entire software industry in the first year.

Andrew S. Tanenbaum, Computer Networks, 2003, Introduction, page 14

Linux registered user #463400

On Sun, Jan 19, 2014 at 08:01:55PM +0100, Matteo Lanati
wrote:
> Hi all,
> 
> I'm running a VM using libvirt+KVM and I have a disk performance issue.
> 
> The host is the following:
> 4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy,
> libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2.
> 
> The guest:
> 1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format.
> 
> When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40"
I get
> around 500 MB/s on bare metal, while only around 30 MB/s inside the VM.
'dd' is rarely a meaningful performance benchmark.

In particular with the way you're invoking it, you're not requesting
that the OS flush data to disk - so the data will be building up in
RAM which is very fast. Since your host likely has much more free RAM
for I/O cache than the guest, dd in the host is able todo much more
I/O to fast RAM before the OS has to start actually writing to slow
disk. The guest which is likely writing to slow disk much sooner giving
the lower results.

At the very least you need to request use of O_DIRECT with dd
so it is writing straight to disk. Or try a formal benchmark tool
like iozone

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Hey,

First of all, you don't need to set scheduler as deadline on VM. Just
disable it (elevator=none). The HV will handle it for the VM.

Furthermore, as generic performance optimization, I recommend you
disable swappiness on the HV, as well as zone reclaim.

I invite you to read this IBM paper:
http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaat/liaattunkickoff.htm
It's quite a good starting point.

Regards,

On 01/19/2014 08:01 PM, Matteo Lanati wrote:
> Hi all,
>
> I'm running a VM using libvirt+KVM and I have a disk performance issue.
>
> The host is the following:
> 4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy,
> libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2.
>
> The guest:
> 1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format.
>
> When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40"
I get
> around 500 MB/s on bare metal, while only around 30 MB/s inside the VM.
>
> I'm trying to get something more out of the virtualization layer, I
hope
> that there's room for improvement.
>
> I'm using virtio, I aready set cache='none' and
io='native' in the domain
> definition. Both host and guest are using deadline as I/O scheduler. The VM
> uses an ext4 filesystem, while the image is saved on an ext3 disk. I
> mounted the host and guest filesystems specifying nodiratime and noatime
> options. Even if I convert the image to raw format nothing changes.
> I didn't mess with iotune nor blockio.
>
> Is there something that I overlooked or any other suggestion?
>
> Thanks in advance for your help.
>
> Matteo
>
>
>
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users

-- 
Pierre Schweitzer <pierre@reactos.org>
System Administrator
ReactOS Foundation

Hi Daniel,

thank you very much for your explanation.
Now every fits into the picture. I run the tests you suggested and the numbers
make much more sense.
Thanks for stating the obvious but it was the first time I needed to compare
I/O.
Best regards,

Matteo


On 20 Jan 2014, at 15:19, Daniel P. Berrange <berrange@redhat.com> wrote:
> 'dd' is rarely a meaningful performance benchmark.
> 
> In particular with the way you're invoking it, you're not
requesting
> that the OS flush data to disk - so the data will be building up in
> RAM which is very fast. Since your host likely has much more free RAM
> for I/O cache than the guest, dd in the host is able todo much more
> I/O to fast RAM before the OS has to start actually writing to slow
> disk. The guest which is likely writing to slow disk much sooner giving
> the lower results.
> 
> At the very least you need to request use of O_DIRECT with dd
> so it is writing straight to disk. Or try a formal benchmark tool
> like iozone
> 
> Daniel
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
:|
> |: http://libvirt.org              -o-             http://virt-manager.org
:|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
:|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
:|
A refund for defective software might be nice, except it would bankrupt the
entire software industry in the first year.

Andrew S. Tanenbaum, Computer Networks, 2003, Introduction, page 14

Linux registered user #463400