Xen devel - Dec 2010 - qemu VS tapdisk2 VS blkback benchmarks

Hi all,
as promised I run some simple benchmarks using tapdisk2 and the new qemu
as disk backends.
If you are in a hurry skip to the last section to see the test results.



SOFTWARE REQUIREMENTS
---------------------
In order to reproduce these results you need to use the new qemu with
linux aio and O_DIRECT as disk backend:

- apply the libxl patches that Anthony sent to the list a little while ago;

- compile qemu with linux aio support, you might need few hacks to work
around limitations of the glic/libaio installed in your system;

- add BDRV_O_NOCACHE|BDRV_O_NATIVE_AIO to the flags used by qemu to open
the disks;

- some gntdev fixes to allow aio and O_DIRECT on granted pages, not yet
sent to the list (but soon).




TEST HARDWARE
-------------
I am using a not so new testbox with a 64bit 2.6.37 dom0 with 752MB of ram.
The guest is a 64 bit PV Lenny guest with a 2.6.37 upstream kernel and 512MB of
ram.
Other info about the host follows:

kontiki:~# cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Genuine Intel(R) CPU 3.00GHz
stepping        : 2
cpu MHz         : 3000.014
cache size      : 2048 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cid cx16
hypervisor lahf_lm arat
bogomips        : 6000.02
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Genuine Intel(R) CPU 3.00GHz
stepping        : 2
cpu MHz         : 3000.014
cache size      : 2048 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cid cx16
hypervisor lahf_lm arat
bogomips        : 6000.02
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:


kontiki:~# smartctl -a /dev/sda
Model Family:     Hitachi Deskstar T7K250 series
Device Model:     HDT722516DLA380
Serial Number:    VDN71BTCDDVR6G
Firmware Version: V43OA91A
User Capacity:    164,696,555,520 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
Local Time is:    Fri Dec 10 16:07:57 2010 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled






TEST RESULTS
------------
The test is simple: write 1GB of data to disk and measure bandwidth
and cpu usage.

- tapdisk2 on raw file
bandwidth: 32MB/s
average cpu usage: 22%

- qemu on raw file
bandwidth: 33MB/s
average cpu usage: 12%

- blkback on LVM
bandwidth: 39MB/s

- qemu on LVM
bandwidth: 38MB/s




CONCLUSIONS
-----------
Qemu beats tapdisk2 on raw files (the bandwidth is the same but the cpu
usage is lower).
Qemu has similar performances to blkback on LVM from the bandwidth
perspective, but I didn''t measure the cpu usage in that case.


Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 2010-12-10 at 11:45 -0500, Stefano Stabellini
wrote:
> Hi all,
> as promised I run some simple benchmarks using tapdisk2 and the new qemu
> as disk backends.
> If you are in a hurry skip to the last section to see the test results.
Interesting. I''ll look into it when the userspace stuff is working.

Thanks!

Daniel
> 
> SOFTWARE REQUIREMENTS
> ---------------------
> In order to reproduce these results you need to use the new qemu with
> linux aio and O_DIRECT as disk backend:
> 
> - apply the libxl patches that Anthony sent to the list a little while ago;
> 
> - compile qemu with linux aio support, you might need few hacks to work
> around limitations of the glic/libaio installed in your system;
> 
> - add BDRV_O_NOCACHE|BDRV_O_NATIVE_AIO to the flags used by qemu to open
> the disks;
> 
> - some gntdev fixes to allow aio and O_DIRECT on granted pages, not yet
> sent to the list (but soon).
> 
> 
> 
> 
> TEST HARDWARE
> -------------
> I am using a not so new testbox with a 64bit 2.6.37 dom0 with 752MB of ram.
> The guest is a 64 bit PV Lenny guest with a 2.6.37 upstream kernel and
512MB of ram.
> Other info about the host follows:
> 
> kontiki:~# cat /proc/cpuinfo 
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 15
> model           : 6
> model name      : Genuine Intel(R) CPU 3.00GHz
> stepping        : 2
> cpu MHz         : 3000.014
> cache size      : 2048 KB
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 6
> wp              : yes
> flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cid cx16
hypervisor lahf_lm arat
> bogomips        : 6000.02
> clflush size    : 64
> cache_alignment : 128
> address sizes   : 36 bits physical, 48 bits virtual
> power management:
> 
> processor       : 1
> vendor_id       : GenuineIntel
> cpu family      : 15
> model           : 6
> model name      : Genuine Intel(R) CPU 3.00GHz
> stepping        : 2
> cpu MHz         : 3000.014
> cache size      : 2048 KB
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 6
> wp              : yes
> flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cid cx16
hypervisor lahf_lm arat
> bogomips        : 6000.02
> clflush size    : 64
> cache_alignment : 128
> address sizes   : 36 bits physical, 48 bits virtual
> power management:
> 
> 
> kontiki:~# smartctl -a /dev/sda
> Model Family:     Hitachi Deskstar T7K250 series
> Device Model:     HDT722516DLA380
> Serial Number:    VDN71BTCDDVR6G
> Firmware Version: V43OA91A
> User Capacity:    164,696,555,520 bytes
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   7
> ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
> Local Time is:    Fri Dec 10 16:07:57 2010 GMT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> 
> 
> 
> 
> 
> 
> TEST RESULTS
> ------------
> The test is simple: write 1GB of data to disk and measure bandwidth
> and cpu usage.
> 
> - tapdisk2 on raw file
> bandwidth: 32MB/s
> average cpu usage: 22%
> 
> - qemu on raw file
> bandwidth: 33MB/s
> average cpu usage: 12%
> 
> - blkback on LVM
> bandwidth: 39MB/s
> 
> - qemu on LVM
> bandwidth: 38MB/s
> 
> 
> 
> 
> CONCLUSIONS
> -----------
> Qemu beats tapdisk2 on raw files (the bandwidth is the same but the cpu
> usage is lower).
> Qemu has similar performances to blkback on LVM from the bandwidth
> perspective, but I didn''t measure the cpu usage in that case.
> 
> 
> Cheers,
> 
> Stefano
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 10 Dec 2010, Daniel Stodden wrote:
> On Fri, 2010-12-10 at 11:45 -0500, Stefano Stabellini wrote:
> > Hi all,
> > as promised I run some simple benchmarks using tapdisk2 and the new
qemu
> > as disk backends.
> > If you are in a hurry skip to the last section to see the test
results.
> 
> Interesting. I''ll look into it when the userspace stuff is
working.
> 
I have a patch series almost ready to fix aio/O_DIRECT from userspace,
I''ll post it early next week.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I am assuming when you said "blkback over lvm", you
used "phy:/dev/..." format.
do you have any figures on tapdisk2 over lvm?
[wherein you would use "tap2:aio:/dev/..." ]

shriram
-- 
perception is but an offspring of its own self

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 11 Dec 2010, Shriram Rajagopalan wrote:
> I am assuming when you said "blkback over lvm", you
> used "phy:/dev/..." format.
> do you have any figures on tapdisk2 over lvm?
> [wherein you would use "tap2:aio:/dev/..." ]
About 37MB/s.

However both tapdisk2 and qemu performances on lvm seem to degrade after
the first 100MB of data transfer to less than 30 MB/s, not sure why.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel