Hi,
I experience massive I/O performance-difference between Dom0 and DomU. To simply
show the impact, I use a logical volume striped over 6 hard-drives. Then
(write/read) enough bytes linear via dd zeros (on/from) this lv and compare the
time.
The result: inside DomU I get only about 30% of the I/O performance of the Dom0.
So, where is the handbrake, and how do I release it? Normaly there shouldbe no
such big difference. If you need more information about setup and config, to
give me a hint where else to search for my missing 70% performance, simply reply
inside the list.
Here are the gory details so far:
Preparations
=================================================================================================Dom0:
akaar
DomU: guinan
Create LV:
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Attach LV to domU:
root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w
Test:
==================================================================================================================
Write
=== Dom0
root@akaar:~# date ; dd if=/dev/zero of=/dev/vg0/performance-test bs=1M
count=4000 ; sync ; date
Do 5. Apr 20:15:50 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 7,34972 s, 571 MB/s
Do 5. Apr 20:15:58 CEST 2012
=== DomU
root@guinan:~# date ; dd if=/dev/zero of=/dev/xvdc1 bs=1M count=4000 ; sync ;
date
Do 5. Apr 20:34:50 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 23,5331 s, 178 MB/s
Do 5. Apr 20:35:13 CEST 2012
============== Read
=== Dom0
root@akaar:~# date ; dd if=/dev/vg0/performance-test of=/dev/null bs=1M
count=4000 ; sync ; date
Do 5. Apr 20:21:38 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 8,98257 s, 467 MB/s
Do 5. Apr 20:21:48 CEST 2012
=== DomU
root@guinan:~# date ; dd if=/dev/xvdc1 of=/dev/null bs=1M count=4000 ; sync ;
date
Do 5. Apr 20:38:04 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 17,8235 s, 235 MB/s
Do 5. Apr 20:38:22 CEST 2012
==================================================================================================================================================================================================
=================================================================================================Technical
Data:
==================================================================================================Hardware
Poweredge 510 / 96GB RAM / 2x X5650
Storage 6 Nearline SAS Harddrives attached to Dell Perc H200:
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Dom0
==================================================================================================
Default Debian Squeeze 32 Bit Kernel 64Bit HV:
1GB Ram / fixed / ballooning disabled / all cpus / no pinning
root@akaar:~# uname -a
Linux akaar 2.6.32-5-xen-686 #1 SMP Mon Mar 26 10:32:49 UTC 2012 i686 GNU/Linux
root@akaar:~# xm info
host : akaar
release : 2.6.32-5-xen-686
version : #1 SMP Mon Mar 26 10:32:49 UTC 2012
machine : i686
nr_cpus : 24
nr_nodes : 2
cores_per_socket : 6
threads_per_core : 2
cpu_mhz : 2666
hw_caps :
bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 98291
free_memory : 95010
node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22
node1:1,3,5,7,9,11,13,15,17,19,21,23
node_to_memory : node0:47110
node1:47899
node_to_dma32_mem : node0:0
node1:2990
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xf9c00000
xen_changeset : unavailable
xen_commandline : placeholder dom0_mem=1024M
cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8)
cc_compile_by : waldi
cc_compile_domain : debian.org
cc_compile_date : Mon Nov 7 09:16:11 CET 2011
xend_config_format : 4
root@akaar:~# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 997 24 r----- 1080.2
guinan 2 4096 4 -b---- 0.0
root@akaar:~# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 20 -b- 747.4 any cpu
Domain-0 0 1 17 -b- 80.2 any cpu
Domain-0 0 2 12 -b- 105.4 any cpu
Domain-0 0 3 19 -b- 49.0 any cpu
Domain-0 0 4 4 -b- 30.7 any cpu
Domain-0 0 5 13 -b- 35.9 any cpu
Domain-0 0 6 21 -b- 52.3 any cpu
Domain-0 0 7 23 -b- 37.9 any cpu
Domain-0 0 8 2 -b- 26.9 any cpu
Domain-0 0 9 7 -b- 32.7 any cpu
Domain-0 0 10 14 -b- 34.8 any cpu
Domain-0 0 11 16 -b- 26.2 any cpu
Domain-0 0 12 10 -b- 23.7 any cpu
Domain-0 0 13 8 -b- 38.7 any cpu
Domain-0 0 14 5 -b- 34.8 any cpu
Domain-0 0 15 11 -b- 30.3 any cpu
Domain-0 0 16 3 -b- 29.3 any cpu
Domain-0 0 17 18 -b- 41.2 any cpu
Domain-0 0 18 14 -b- 41.4 any cpu
Domain-0 0 19 1 -b- 45.6 any cpu
Domain-0 0 20 6 r-- 43.6 any cpu
Domain-0 0 21 15 -b- 29.2 any cpu
Domain-0 0 22 0 -b- 23.2 any cpu
Domain-0 0 23 7 -b- 30.5 any cpu
guinan 2 0 9 -b- 72.1 any cpu
guinan 2 1 15 -b- 2.9 any cpu
guinan 2 2 19 -b- 7.4 any cpu
guinan 2 3 13 -b- 6.0 any cpu
========================================================================================================
DomU
========================================================================================================debian
Squeeze 64Bit 1GB RAM / 4 vcpus / no pinning:
root@guinan:~# uname -a
Linux guinan 2.6.32-5-xen-amd64 #1 SMP Thu Mar 22 21:14:26 UTC 2012 x86_64
GNU/Linux
root@guinan:~# free
total used free shared buffers cached
Mem: 1049180 227444 821736 0 20060 73712
-/+ buffers/cache: 133672 915508
Swap: 2101176 0 2101176
Attach Storage to DomU guinan
root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Cheers
Ulli
--
Ulrich Hochholdinger
Bereichsleiter Infrastruktur
Netpioneer GmbH, Beiertheimer Allee 18, 76137 Karlsruhe
Geschäftsführer: Jürgen Diller
HRB: 107521 Handelsregister Mannheim
Tel: 0721 / 9 20 60 90
Fax: 0721 / 9 20 60 30
E-Mail: Ulrich.Hochholdinger@netpioneer.de
www: http://www.netpioneer.de
Hi Ulli,
I have recently been doing some disk performance benchmarking of various Xen
configurations using HVMs, PVHVMs and PVMs. I have been using the bonnie++
benchmark to do the testing and have come up with pretty much the same results
you have. HVMs are about 25% of the performance of the native machine, PVHVMs
are about 30-35% of the native performance and PVMs are around 50-60% of the
native performance. This is when doing the block reads and writes (character
reads and writes have performance numbers in the 85-95% range).
I would also like to know why this is. Supposedly disk IO with PVHVMs and PVMs
should be close to native performance according to graphs you can find on the
internet, however this does not line up with the actual performance I have been
experiencing.
Several threads on this mailing list recommend people to use PVHVMs as they
should see massive performance increases. Is there a particular workload that it
has a meaningful impact when switching to PVHVMs? So far in my testing it seems
like Xen cannot be used if you require disk intensive operations.
Matt Keeler
-----Original Message-----
From: xen-users-bounces@lists.xen.org [mailto:xen-users-bounces@lists.xen.org]
On Behalf Of Ulrich Hochholdinger
Sent: Friday, April 06, 2012 6:15 AM
To: xen-users@lists.xen.org
Subject: [Xen-users] Disk-I/O Performance DomU vs Dom0
Hi,
I experience massive I/O performance-difference between Dom0 and DomU. To simply
show the impact, I use a logical volume striped over 6 hard-drives. Then
(write/read) enough bytes linear via dd zeros (on/from) this lv and compare the
time.
The result: inside DomU I get only about 30% of the I/O performance of the Dom0.
So, where is the handbrake, and how do I release it? Normaly there shouldbe no
such big difference. If you need more information about setup and config, to
give me a hint where else to search for my missing 70% performance, simply reply
inside the list.
Here are the gory details so far:
Preparations
=================================================================================================Dom0:
akaar
DomU: guinan
Create LV:
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Attach LV to domU:
root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w
Test:
==================================================================================================================
Write
=== Dom0
root@akaar:~# date ; dd if=/dev/zero of=/dev/vg0/performance-test bs=1M
count=4000 ; sync ; date
Do 5. Apr 20:15:50 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 7,34972 s, 571 MB/s
Do 5. Apr 20:15:58 CEST 2012
=== DomU
root@guinan:~# date ; dd if=/dev/zero of=/dev/xvdc1 bs=1M count=4000 ; sync ;
date
Do 5. Apr 20:34:50 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 23,5331 s, 178 MB/s
Do 5. Apr 20:35:13 CEST 2012
============== Read
=== Dom0
root@akaar:~# date ; dd if=/dev/vg0/performance-test of=/dev/null bs=1M
count=4000 ; sync ; date
Do 5. Apr 20:21:38 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 8,98257 s, 467 MB/s
Do 5. Apr 20:21:48 CEST 2012
=== DomU
root@guinan:~# date ; dd if=/dev/xvdc1 of=/dev/null bs=1M count=4000 ; sync ;
date
Do 5. Apr 20:38:04 CEST 2012
4000+0 records in
4000+0 records out
4194304000 bytes (4,2 GB) copied, 17,8235 s, 235 MB/s
Do 5. Apr 20:38:22 CEST 2012
==================================================================================================================================================================================================
=================================================================================================Technical
Data:
==================================================================================================Hardware
Poweredge 510 / 96GB RAM / 2x X5650
Storage 6 Nearline SAS Harddrives attached to Dell Perc H200:
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Dom0
==================================================================================================
Default Debian Squeeze 32 Bit Kernel 64Bit HV:
1GB Ram / fixed / ballooning disabled / all cpus / no pinning
root@akaar:~# uname -a
Linux akaar 2.6.32-5-xen-686 #1 SMP Mon Mar 26 10:32:49 UTC 2012 i686 GNU/Linux
root@akaar:~# xm info
host : akaar
release : 2.6.32-5-xen-686
version : #1 SMP Mon Mar 26 10:32:49 UTC 2012
machine : i686
nr_cpus : 24
nr_nodes : 2
cores_per_socket : 6
threads_per_core : 2
cpu_mhz : 2666
hw_caps :
bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 98291
free_memory : 95010
node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22
node1:1,3,5,7,9,11,13,15,17,19,21,23
node_to_memory : node0:47110
node1:47899
node_to_dma32_mem : node0:0
node1:2990
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xf9c00000
xen_changeset : unavailable
xen_commandline : placeholder dom0_mem=1024M
cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8)
cc_compile_by : waldi
cc_compile_domain : debian.org
cc_compile_date : Mon Nov 7 09:16:11 CET 2011
xend_config_format : 4
root@akaar:~# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 997 24 r----- 1080.2
guinan 2 4096 4 -b---- 0.0
root@akaar:~# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 20 -b- 747.4 any cpu
Domain-0 0 1 17 -b- 80.2 any cpu
Domain-0 0 2 12 -b- 105.4 any cpu
Domain-0 0 3 19 -b- 49.0 any cpu
Domain-0 0 4 4 -b- 30.7 any cpu
Domain-0 0 5 13 -b- 35.9 any cpu
Domain-0 0 6 21 -b- 52.3 any cpu
Domain-0 0 7 23 -b- 37.9 any cpu
Domain-0 0 8 2 -b- 26.9 any cpu
Domain-0 0 9 7 -b- 32.7 any cpu
Domain-0 0 10 14 -b- 34.8 any cpu
Domain-0 0 11 16 -b- 26.2 any cpu
Domain-0 0 12 10 -b- 23.7 any cpu
Domain-0 0 13 8 -b- 38.7 any cpu
Domain-0 0 14 5 -b- 34.8 any cpu
Domain-0 0 15 11 -b- 30.3 any cpu
Domain-0 0 16 3 -b- 29.3 any cpu
Domain-0 0 17 18 -b- 41.2 any cpu
Domain-0 0 18 14 -b- 41.4 any cpu
Domain-0 0 19 1 -b- 45.6 any cpu
Domain-0 0 20 6 r-- 43.6 any cpu
Domain-0 0 21 15 -b- 29.2 any cpu
Domain-0 0 22 0 -b- 23.2 any cpu
Domain-0 0 23 7 -b- 30.5 any cpu
guinan 2 0 9 -b- 72.1 any cpu
guinan 2 1 15 -b- 2.9 any cpu
guinan 2 2 19 -b- 7.4 any cpu
guinan 2 3 13 -b- 6.0 any cpu
========================================================================================================
DomU
========================================================================================================debian
Squeeze 64Bit 1GB RAM / 4 vcpus / no pinning:
root@guinan:~# uname -a
Linux guinan 2.6.32-5-xen-amd64 #1 SMP Thu Mar 22 21:14:26 UTC 2012 x86_64
GNU/Linux
root@guinan:~# free
total used free shared buffers cached
Mem: 1049180 227444 821736 0 20060 73712
-/+ buffers/cache: 133672 915508
Swap: 2101176 0 2101176
Attach Storage to DomU guinan
root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w
lvcreate --name=performance-test --size=20G --stripes=6 vg0
Cheers
Ulli
--
Ulrich Hochholdinger
Bereichsleiter Infrastruktur
Netpioneer GmbH, Beiertheimer Allee 18, 76137 Karlsruhe
Geschäftsführer: Jürgen Diller
HRB: 107521 Handelsregister Mannheim
Tel: 0721 / 9 20 60 90
Fax: 0721 / 9 20 60 30
E-Mail: Ulrich.Hochholdinger@netpioneer.de
www: http://www.netpioneer.de
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Hi, what says dom0 dmesg about disk cache (write, read) of each disk behind Perc 200? We had similar problem, we followed instructions at http://blog.slucas.fr/en/tips/esxi-perc-h200-slow and then the performance dom0 vs. domU was close the same. (but our real numbers were much lower, ? only two disks mirror) Regards K. Dne 6.4.2012 15:38, Keeler, Matthew J. napsal(a):> Hi Ulli, > > I have recently been doing some disk performance benchmarking of various Xen configurations using HVMs, PVHVMs and PVMs. I have been using the bonnie++ benchmark to do the testing and have come up with pretty much the same results you have. HVMs are about 25% of the performance of the native machine, PVHVMs are about 30-35% of the native performance and PVMs are around 50-60% of the native performance. This is when doing the block reads and writes (character reads and writes have performance numbers in the 85-95% range). > > I would also like to know why this is. Supposedly disk IO with PVHVMs and PVMs should be close to native performance according to graphs you can find on the internet, however this does not line up with the actual performance I have been experiencing. > > Several threads on this mailing list recommend people to use PVHVMs as they should see massive performance increases. Is there a particular workload that it has a meaningful impact when switching to PVHVMs? So far in my testing it seems like Xen cannot be used if you require disk intensive operations. > > Matt Keeler > > -----Original Message----- > From: xen-users-bounces@lists.xen.org [mailto:xen-users-bounces@lists.xen.org] On Behalf Of Ulrich Hochholdinger > Sent: Friday, April 06, 2012 6:15 AM > To: xen-users@lists.xen.org > Subject: [Xen-users] Disk-I/O Performance DomU vs Dom0 > > Hi, > I experience massive I/O performance-difference between Dom0 and DomU. To simply show the impact, I use a logical volume striped over 6 hard-drives. Then (write/read) enough bytes linear via dd zeros (on/from) this lv and compare the time. > The result: inside DomU I get only about 30% of the I/O performance of the Dom0. > > So, where is the handbrake, and how do I release it? Normaly there shouldbe no such big difference. If you need more information about setup and config, to give me a hint where else to search for my missing 70% performance, simply reply inside the list. > > Here are the gory details so far: > > Preparations > =================================================================================================> Dom0: akaar > DomU: guinan > > Create LV: > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Attach LV to domU: > root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w > > Test: > ====================================================================================================> ============== Write > === Dom0 > root@akaar:~# date ; dd if=/dev/zero of=/dev/vg0/performance-test bs=1M count=4000 ; sync ; date > Do 5. Apr 20:15:50 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 7,34972 s, 571 MB/s > Do 5. Apr 20:15:58 CEST 2012 > > === DomU > root@guinan:~# date ; dd if=/dev/zero of=/dev/xvdc1 bs=1M count=4000 ; sync ; date > Do 5. Apr 20:34:50 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 23,5331 s, 178 MB/s > Do 5. Apr 20:35:13 CEST 2012 > > > ============== Read > === Dom0 > root@akaar:~# date ; dd if=/dev/vg0/performance-test of=/dev/null bs=1M count=4000 ; sync ; date > Do 5. Apr 20:21:38 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 8,98257 s, 467 MB/s > Do 5. Apr 20:21:48 CEST 2012 > > === DomU > root@guinan:~# date ; dd if=/dev/xvdc1 of=/dev/null bs=1M count=4000 ; sync ; date > Do 5. Apr 20:38:04 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 17,8235 s, 235 MB/s > Do 5. Apr 20:38:22 CEST 2012 > =================================================================================================> =================================================================================================> > > =================================================================================================> Technical Data: > ==================================================================================================> Hardware Poweredge 510 / 96GB RAM / 2x X5650 > > Storage 6 Nearline SAS Harddrives attached to Dell Perc H200: > > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Dom0 > ==================================================================================================> Default Debian Squeeze 32 Bit Kernel 64Bit HV: > 1GB Ram / fixed / ballooning disabled / all cpus / no pinning > > root@akaar:~# uname -a > Linux akaar 2.6.32-5-xen-686 #1 SMP Mon Mar 26 10:32:49 UTC 2012 i686 GNU/Linux > > > root@akaar:~# xm info > host : akaar > release : 2.6.32-5-xen-686 > version : #1 SMP Mon Mar 26 10:32:49 UTC 2012 > machine : i686 > nr_cpus : 24 > nr_nodes : 2 > cores_per_socket : 6 > threads_per_core : 2 > cpu_mhz : 2666 > hw_caps : bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 98291 > free_memory : 95010 > node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 > node1:1,3,5,7,9,11,13,15,17,19,21,23 > node_to_memory : node0:47110 > node1:47899 > node_to_dma32_mem : node0:0 > node1:2990 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .1 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xf9c00000 > xen_changeset : unavailable > xen_commandline : placeholder dom0_mem=1024M > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) > cc_compile_by : waldi > cc_compile_domain : debian.org > cc_compile_date : Mon Nov 7 09:16:11 CET 2011 > xend_config_format : 4 > > root@akaar:~# xm list > Name ID Mem VCPUs State Time(s) > Domain-0 0 997 24 r----- 1080.2 > guinan 2 4096 4 -b---- 0.0 > > root@akaar:~# xm vcpu-list > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 20 -b- 747.4 any cpu > Domain-0 0 1 17 -b- 80.2 any cpu > Domain-0 0 2 12 -b- 105.4 any cpu > Domain-0 0 3 19 -b- 49.0 any cpu > Domain-0 0 4 4 -b- 30.7 any cpu > Domain-0 0 5 13 -b- 35.9 any cpu > Domain-0 0 6 21 -b- 52.3 any cpu > Domain-0 0 7 23 -b- 37.9 any cpu > Domain-0 0 8 2 -b- 26.9 any cpu > Domain-0 0 9 7 -b- 32.7 any cpu > Domain-0 0 10 14 -b- 34.8 any cpu > Domain-0 0 11 16 -b- 26.2 any cpu > Domain-0 0 12 10 -b- 23.7 any cpu > Domain-0 0 13 8 -b- 38.7 any cpu > Domain-0 0 14 5 -b- 34.8 any cpu > Domain-0 0 15 11 -b- 30.3 any cpu > Domain-0 0 16 3 -b- 29.3 any cpu > Domain-0 0 17 18 -b- 41.2 any cpu > Domain-0 0 18 14 -b- 41.4 any cpu > Domain-0 0 19 1 -b- 45.6 any cpu > Domain-0 0 20 6 r-- 43.6 any cpu > Domain-0 0 21 15 -b- 29.2 any cpu > Domain-0 0 22 0 -b- 23.2 any cpu > Domain-0 0 23 7 -b- 30.5 any cpu > guinan 2 0 9 -b- 72.1 any cpu > guinan 2 1 15 -b- 2.9 any cpu > guinan 2 2 19 -b- 7.4 any cpu > guinan 2 3 13 -b- 6.0 any cpu > ========================================================================================================> > > DomU > ========================================================================================================> debian Squeeze 64Bit 1GB RAM / 4 vcpus / no pinning: > > root@guinan:~# uname -a > Linux guinan 2.6.32-5-xen-amd64 #1 SMP Thu Mar 22 21:14:26 UTC 2012 x86_64 GNU/Linux > > root@guinan:~# free > total used free shared buffers cached > Mem: 1049180 227444 821736 0 20060 73712 > -/+ buffers/cache: 133672 915508 > Swap: 2101176 0 2101176 > > > > Attach Storage to DomU guinan > root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w > > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Cheers > Ulli > -- > Ulrich Hochholdinger > Bereichsleiter Infrastruktur >
On 6 April 2012 20:15, Ulrich Hochholdinger <uhochholdinger@netpioneer.de> wrote:> Hi, > I experience massive I/O performance-difference between Dom0 and DomU. To simply show the impact, I use a logical volume striped over 6 hard-drives. Then (write/read) enough bytes linear via dd zeros (on/from) this lv and compare the time. > The result: inside DomU I get only about 30% of the I/O performance of the Dom0. > > So, where is the handbrake, and how do I release it? Normaly there shouldbe no such big difference. If you need more information about setup and config, to give me a hint where else to search for my missing 70% performance, simply reply inside the list. > > Here are the gory details so far: > > Preparations > =================================================================================================> Dom0: akaar > DomU: guinan > > Create LV: > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Attach LV to domU: > root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w > > Test: > ====================================================================================================> ============== Write > === Dom0 > root@akaar:~# date ; dd if=/dev/zero of=/dev/vg0/performance-test bs=1M count=4000 ; sync ; date > Do 5. Apr 20:15:50 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 7,34972 s, 571 MB/s > Do 5. Apr 20:15:58 CEST 2012 > > === DomU > root@guinan:~# date ; dd if=/dev/zero of=/dev/xvdc1 bs=1M count=4000 ; sync ; date > Do 5. Apr 20:34:50 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 23,5331 s, 178 MB/s > Do 5. Apr 20:35:13 CEST 2012 > > > ============== Read > === Dom0 > root@akaar:~# date ; dd if=/dev/vg0/performance-test of=/dev/null bs=1M count=4000 ; sync ; date > Do 5. Apr 20:21:38 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 8,98257 s, 467 MB/s > Do 5. Apr 20:21:48 CEST 2012 > > === DomU > root@guinan:~# date ; dd if=/dev/xvdc1 of=/dev/null bs=1M count=4000 ; sync ; date > Do 5. Apr 20:38:04 CEST 2012 > 4000+0 records in > 4000+0 records out > 4194304000 bytes (4,2 GB) copied, 17,8235 s, 235 MB/s > Do 5. Apr 20:38:22 CEST 2012 > =================================================================================================> =================================================================================================> > > =================================================================================================> Technical Data: > ==================================================================================================> Hardware Poweredge 510 / 96GB RAM / 2x X5650 > > Storage 6 Nearline SAS Harddrives attached to Dell Perc H200: > > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Dom0 > ==================================================================================================> Default Debian Squeeze 32 Bit Kernel 64Bit HV: > 1GB Ram / fixed / ballooning disabled / all cpus / no pinning > > root@akaar:~# uname -a > Linux akaar 2.6.32-5-xen-686 #1 SMP Mon Mar 26 10:32:49 UTC 2012 i686 GNU/Linux > > > root@akaar:~# xm info > host : akaar > release : 2.6.32-5-xen-686 > version : #1 SMP Mon Mar 26 10:32:49 UTC 2012 > machine : i686 > nr_cpus : 24 > nr_nodes : 2 > cores_per_socket : 6 > threads_per_core : 2 > cpu_mhz : 2666 > hw_caps : bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 98291 > free_memory : 95010 > node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 > node1:1,3,5,7,9,11,13,15,17,19,21,23 > node_to_memory : node0:47110 > node1:47899 > node_to_dma32_mem : node0:0 > node1:2990 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .1 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xf9c00000 > xen_changeset : unavailable > xen_commandline : placeholder dom0_mem=1024M > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) > cc_compile_by : waldi > cc_compile_domain : debian.org > cc_compile_date : Mon Nov 7 09:16:11 CET 2011 > xend_config_format : 4 > > root@akaar:~# xm list > Name ID Mem VCPUs State Time(s) > Domain-0 0 997 24 r----- 1080.2 > guinan 2 4096 4 -b---- 0.0 > > root@akaar:~# xm vcpu-list > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 20 -b- 747.4 any cpu > Domain-0 0 1 17 -b- 80.2 any cpu > Domain-0 0 2 12 -b- 105.4 any cpu > Domain-0 0 3 19 -b- 49.0 any cpu > Domain-0 0 4 4 -b- 30.7 any cpu > Domain-0 0 5 13 -b- 35.9 any cpu > Domain-0 0 6 21 -b- 52.3 any cpu > Domain-0 0 7 23 -b- 37.9 any cpu > Domain-0 0 8 2 -b- 26.9 any cpu > Domain-0 0 9 7 -b- 32.7 any cpu > Domain-0 0 10 14 -b- 34.8 any cpu > Domain-0 0 11 16 -b- 26.2 any cpu > Domain-0 0 12 10 -b- 23.7 any cpu > Domain-0 0 13 8 -b- 38.7 any cpu > Domain-0 0 14 5 -b- 34.8 any cpu > Domain-0 0 15 11 -b- 30.3 any cpu > Domain-0 0 16 3 -b- 29.3 any cpu > Domain-0 0 17 18 -b- 41.2 any cpu > Domain-0 0 18 14 -b- 41.4 any cpu > Domain-0 0 19 1 -b- 45.6 any cpu > Domain-0 0 20 6 r-- 43.6 any cpu > Domain-0 0 21 15 -b- 29.2 any cpu > Domain-0 0 22 0 -b- 23.2 any cpu > Domain-0 0 23 7 -b- 30.5 any cpu > guinan 2 0 9 -b- 72.1 any cpu > guinan 2 1 15 -b- 2.9 any cpu > guinan 2 2 19 -b- 7.4 any cpu > guinan 2 3 13 -b- 6.0 any cpu > ========================================================================================================> > > DomU > ========================================================================================================> debian Squeeze 64Bit 1GB RAM / 4 vcpus / no pinning: > > root@guinan:~# uname -a > Linux guinan 2.6.32-5-xen-amd64 #1 SMP Thu Mar 22 21:14:26 UTC 2012 x86_64 GNU/Linux > > root@guinan:~# free > total used free shared buffers cached > Mem: 1049180 227444 821736 0 20060 73712 > -/+ buffers/cache: 133672 915508 > Swap: 2101176 0 2101176 > > > > Attach Storage to DomU guinan > root@akaar:~# xm block-attach guinan phy:/dev/vg0/performance-test xvdc1 w > > lvcreate --name=performance-test --size=20G --stripes=6 vg0 > > Cheers > Ulli > -- > Ulrich Hochholdinger > Bereichsleiter Infrastruktur > > Netpioneer GmbH, Beiertheimer Allee 18, 76137 Karlsruhe > Geschäftsführer: Jürgen Diller > HRB: 107521 Handelsregister Mannheim > > Tel: 0721 / 9 20 60 90 > Fax: 0721 / 9 20 60 30 > E-Mail: Ulrich.Hochholdinger@netpioneer.de > www: http://www.netpioneer.de > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-usersHi, I am guesing you are using Debian Squeeze. Pure PV domain performance on the Squeeze pvops dom0 kernel isn''t too bad but definitely isn''t on par with 3.2 pvops dom0 kernels. I would highly suggest not using dd for performance testing unless you write much more than 4GB. Atleast 2x the ammount of RAM should be written to ensure the Linux disk cache isn''t playing with your data. Using highly compressible zeros isn''t really that great either for managing disks that may be behind a raid controller etc. To be frank there is a huge amount to know about properly benchmarking I/O performance and detailing all of it goes beyond the depth of this post but atleast do the following: 1 ) Use the Flexible IO Tester also known as fio. It''s by far the best and most reliable I/O testing tool. 2) Where possible use raw volumes, where not use LVM (You are using LVM which is good) 3) Always write atleast 2x the amount of RAM you have in the dom0, for best results write much more. For instance for a 4GB dom0 when I am doing SSD performance testing I write around 100GB of data. If you want really good I/O performance because it''s important for your workload I would highly recommend using a more recent 3.2 dom0 kernel. I understand this is not practical for some that need to use "Enterprise" distros but that is the price you pay for using old kernels. Debian Squeeze with 2.6.32 is incredibly stable and I would recommend leaving it alone if that''s what matters to you most. Joseph. -- Founder | Director | VP Research Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56 99 52 | Mobile: 0428 754 846
On Mon, Apr 9, 2012 at 12:44 PM, Joseph Glanville <joseph.glanville@orionvm.com.au> wrote:> If you want really good I/O performance because it''s important for > your workload I would highly recommend using a more recent 3.2 dom0 > kernel. > I understand this is not practical for some that need to use > "Enterprise" distros but that is the price you pay for using old > kernels.Some "enterprise" distros and their kernel version: - Oracle Linux 5/6: 3.0.x (when using kernel UEK2). Should be installabe on RHEL/centos as well, but you won''t get the support (obviously) - SLES11 SP2: 3.0.10 - Ubuntu 11.04 (when it''s released): 3.2.x> Debian Squeeze with 2.6.32 is incredibly stable and I would recommend > leaving it alone if that''s what matters to you most.Judging from the mails to this list from debian users, I wouldn''t recommend using their built-in xen+kernel packages. Period. -- Fajar
On 9 April 2012 16:25, Fajar A. Nugraha <list@fajar.net> wrote:> On Mon, Apr 9, 2012 at 12:44 PM, Joseph Glanville > <joseph.glanville@orionvm.com.au> wrote: >> If you want really good I/O performance because it''s important for >> your workload I would highly recommend using a more recent 3.2 dom0 >> kernel. >> I understand this is not practical for some that need to use >> "Enterprise" distros but that is the price you pay for using old >> kernels. > > Some "enterprise" distros and their kernel version: > - Oracle Linux 5/6: 3.0.x (when using kernel UEK2). Should be > installabe on RHEL/centos as well, but you won''t get the support > (obviously) > - SLES11 SP2: 3.0.10 > - Ubuntu 11.04 (when it''s released): 3.2.x > >> Debian Squeeze with 2.6.32 is incredibly stable and I would recommend >> leaving it alone if that''s what matters to you most. > > Judging from the mails to this list from debian users, I wouldn''t > recommend using their built-in xen+kernel packages. Period.The Debian Squeeze setup is fine. It''s mostly user error that results in bad performance and or problems. That being said I have started development of a dom0 livecd using Ubuntu 12.04 which is considerably better. Packing Xen 4.1.2 + 3.2 is a great combination. I will be open-sourcing the build chain on github that produces the livecd.> > -- > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users-- Founder | Director | VP Research Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56 99 52 | Mobile: 0428 754 846