Marcin Owsiany
2007-Apr-17 15:28 UTC
[Xen-users] Big I/O performance difference between dom0 and domU
Hi,
I am setting up a dual CPU PowerEdge 2550 system with PERC 3Di
controller (aacraid) with 3 18GB disks in RAID5 and XEN 3.0.3-0-2,
credit scheduler, PAE (the Debian package in etch). This is not best
hardware, but what worries me more is the poor I/O performance in domU
compared to dom0.
With both domains having 500 MB of RAM, testing with bonnie++ on the
same 5GB LVM volume with xfs filesystem, with 4GB test data size, I''m
getting:
-----+-------------------+--------------------+-----------------------
Test | block read [kB/s] | block write [kB/s] | random seeks [/sec]
-----+-------------------+--------------------+-----------------------
dom0 | 10308 | 64806 | 325.3
domU | 7299 | 53469 | 265.6
-----+-------------------+--------------------+-----------------------
~drop| 30% | 17% | 18%
-----+-------------------+--------------------+-----------------------
The results are basically the same whether I use the default vcpu
arrangement (2 vcpus for dom0, one for domU) or set it to one vcpu per
domain, each pinned to a different physical cpu.
Any suggestions are welcome...
--
Marcin Owsiany <marcin@owsiany.pl> http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
"Every program in development at MIT expands until it can read mail."
-- Unknown
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Liang Yang
2007-Apr-17 16:34 UTC
Re: [Xen-users] Big I/O performance difference between dom0 and domU
What is the CPU utilization when you did this I/O performance measurement? As far I can remember, Bonnie++ does not support using outstanding I/Os. So your target RAID volume may not be saturated, i.e. you did not get the peak performance. Liang ----- Original Message ----- From: "Marcin Owsiany" <marcin@owsiany.pl> To: "Xen-Users" <xen-users@lists.xensource.com> Sent: Tuesday, April 17, 2007 8:28 AM Subject: [Xen-users] Big I/O performance difference between dom0 and domU> Hi, > > I am setting up a dual CPU PowerEdge 2550 system with PERC 3Di > controller (aacraid) with 3 18GB disks in RAID5 and XEN 3.0.3-0-2, > credit scheduler, PAE (the Debian package in etch). This is not best > hardware, but what worries me more is the poor I/O performance in domU > compared to dom0. > > With both domains having 500 MB of RAM, testing with bonnie++ on the > same 5GB LVM volume with xfs filesystem, with 4GB test data size, I''m > getting: > > -----+-------------------+--------------------+----------------------- > Test | block read [kB/s] | block write [kB/s] | random seeks [/sec] > -----+-------------------+--------------------+----------------------- > dom0 | 10308 | 64806 | 325.3 > domU | 7299 | 53469 | 265.6 > -----+-------------------+--------------------+----------------------- > ~drop| 30% | 17% | 18% > -----+-------------------+--------------------+----------------------- > > The results are basically the same whether I use the default vcpu > arrangement (2 vcpus for dom0, one for domU) or set it to one vcpu per > domain, each pinned to a different physical cpu. > > Any suggestions are welcome... > > > -- > Marcin Owsiany <marcin@owsiany.pl> http://marcin.owsiany.pl/ > GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 > > "Every program in development at MIT expands until it can read mail." > -- Unknown > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Marcin Owsiany
2007-Apr-17 16:57 UTC
Re: [Xen-users] Big I/O performance difference between dom0 and domU
On Tue, Apr 17, 2007 at 09:34:10AM -0700, Liang Yang wrote:> What is the CPU utilization when you did this I/O performance measurement?"% CPU" reported by bonnie when writing/reading/random seeking in is 6/3/0% in dom0 and 4/2/0% in domU. bonnie was the only thing running when performing the test.> As far I can remember, Bonnie++ does not support using outstanding I/Os. > So your target RAID volume may not be saturated, i.e. you did not get the > peak performance.What other benchmarking tool would you suggest? -- Marcin Owsiany <marcin@owsiany.pl> http://marcin.owsiany.pl/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 "Every program in development at MIT expands until it can read mail." -- Unknown _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luke S. Crawford
2007-Apr-17 17:57 UTC
Re: [Xen-users] Big I/O performance difference between dom0 and domU
On Tue, 17 Apr 2007, Marcin Owsiany wrote:> On Tue, Apr 17, 2007 at 09:34:10AM -0700, Liang Yang wrote: >> What is the CPU utilization when you did this I/O performance measurement? > > "% CPU" reported by bonnie when writing/reading/random seeking in is > 6/3/0% in dom0 and 4/2/0% in domU. bonnie was the only thing running > when performing the test.watch top in the Dom0 while running bonnie in the DomU; better, watch xentop in the Dom0. when you do I/O in the DomU, it needs to go through the Dom0 in order to get to the real disk. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Marcin Owsiany
2007-Apr-18 09:48 UTC
Re: [Xen-users] Big I/O performance difference between dom0 and domU
On Tue, Apr 17, 2007 at 05:57:02PM +0100, Marcin Owsiany wrote:> On Tue, Apr 17, 2007 at 09:34:10AM -0700, Liang Yang wrote: > > What is the CPU utilization when you did this I/O performance measurement? > > "% CPU" reported by bonnie when writing/reading/random seeking in is > 6/3/0% in dom0 and 4/2/0% in domU. bonnie was the only thing running > when performing the test. > > > As far I can remember, Bonnie++ does not support using outstanding I/Os. > > So your target RAID volume may not be saturated, i.e. you did not get the > > peak performance. > > What other benchmarking tool would you suggest?OK, I ran tiobench, which does a multi-threaded test. The conclusions are basically the same, there is a HUGE performance drop when moving from dom0 to domU. Unit information ===============File size = megabytes Blk Size = bytes Rate = megabytes per second CPU% = percentage of CPU used during the test Latency = milliseconds Lat% = percent of requests that took longer than X seconds CPU Eff = Rate divided by CPU% - throughput per cpu load ====== dom0 ===== Sequential Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1454 4096 1 59.41 4.243% 0.325 136.06 0.00000 0.00000 1400 2.6.18-4-xen-686 1454 4096 2 55.44 7.004% 0.695 640.86 0.00000 0.00000 792 2.6.18-4-xen-686 1454 4096 4 54.86 14.34% 1.403 1794.06 0.00000 0.00000 382 2.6.18-4-xen-686 1454 4096 8 45.94 20.58% 3.231 4126.18 0.00000 0.00000 223 Random Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1454 4096 1 0.94 0.183% 20.700 262.32 0.00000 0.00000 514 2.6.18-4-xen-686 1454 4096 2 1.61 0.049% 23.767 99.26 0.00000 0.00000 3255 2.6.18-4-xen-686 1454 4096 4 2.29 0.328% 32.698 203.58 0.00000 0.00000 698 2.6.18-4-xen-686 1454 4096 8 2.72 0.292% 49.862 574.06 0.00000 0.00000 930 Sequential Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1454 4096 1 10.48 7.397% 1.382 18330.81 0.00243 0.00000 142 2.6.18-4-xen-686 1454 4096 2 10.08 14.59% 2.734 41922.34 0.00994 0.00027 69 2.6.18-4-xen-686 1454 4096 4 9.74 28.07% 5.620 108145.49 0.04493 0.00351 35 2.6.18-4-xen-686 1454 4096 8 9.56 53.08% 10.883 105600.05 0.13138 0.00675 18 Random Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1454 4096 1 1.23 0.819% 0.108 0.55 0.00000 0.00000 150 2.6.18-4-xen-686 1454 4096 2 1.24 1.258% 0.125 40.93 0.00000 0.00000 99 2.6.18-4-xen-686 1454 4096 4 1.22 1.970% 0.125 74.78 0.00000 0.00000 62 2.6.18-4-xen-686 1454 4096 8 1.12 3.463% 0.107 1.53 0.00000 0.00000 32 ====== domU ===== Sequential Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1016 4096 1 47.22 2.435% 0.410 107.34 0.00000 0.00000 1939 2.6.18-4-xen-686 1016 4096 2 50.51 6.466% 0.768 149.37 0.00000 0.00000 781 2.6.18-4-xen-686 1016 4096 4 34.17 6.062% 2.270 324.78 0.00000 0.00000 564 2.6.18-4-xen-686 1016 4096 8 30.68 10.90% 5.011 843.47 0.00000 0.00000 281 Random Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1016 4096 1 1.39 0.021% 14.048 68.12 0.00000 0.00000 6510 2.6.18-4-xen-686 1016 4096 2 1.42 0.029% 27.293 107.90 0.00000 0.00000 4883 2.6.18-4-xen-686 1016 4096 4 2.86 0.351% 26.259 210.81 0.00000 0.00000 814 2.6.18-4-xen-686 1016 4096 8 4.08 0.334% 35.834 401.12 0.00000 0.00000 1221 Sequential Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1016 4096 1 6.98 4.370% 2.223 8511.98 0.00000 0.00000 160 2.6.18-4-xen-686 1016 4096 2 6.00 7.388% 4.795 18369.97 0.07612 0.00000 81 2.6.18-4-xen-686 1016 4096 4 5.88 13.93% 9.530 39506.77 0.22645 0.00000 42 2.6.18-4-xen-686 1016 4096 8 5.27 22.37% 16.923 79262.92 0.26606 0.01999 24 Random Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.6.18-4-xen-686 1016 4096 1 1.01 0.524% 0.101 2.45 0.00000 0.00000 193 2.6.18-4-xen-686 1016 4096 2 1.03 0.975% 0.127 85.29 0.00000 0.00000 106 2.6.18-4-xen-686 1016 4096 4 0.98 1.461% 0.102 2.79 0.00000 0.00000 67 2.6.18-4-xen-686 1016 4096 8 0.93 2.610% 0.101 2.49 0.00000 0.00000 36 -- Marcin Owsiany <marcin@owsiany.pl> http://marcin.owsiany.pl/ GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216 "Every program in development at MIT expands until it can read mail." -- Unknown _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-Apr-18 10:17 UTC
RE: [Xen-users] Big I/O performance difference between dom0 and domU
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Marcin Owsiany > Sent: 18 April 2007 10:48 > To: xen-users@lists.xensource.com > Cc: Liang Yang > Subject: Re: [Xen-users] Big I/O performance difference > between dom0 and domU > > On Tue, Apr 17, 2007 at 05:57:02PM +0100, Marcin Owsiany wrote: > > On Tue, Apr 17, 2007 at 09:34:10AM -0700, Liang Yang wrote: > > > What is the CPU utilization when you did this I/O > performance measurement? > > > > "% CPU" reported by bonnie when writing/reading/random seeking in is > > 6/3/0% in dom0 and 4/2/0% in domU. bonnie was the only thing running > > when performing the test. > > > > > As far I can remember, Bonnie++ does not support using > outstanding I/Os. > > > So your target RAID volume may not be saturated, i.e. you > did not get the > > > peak performance. > > > > What other benchmarking tool would you suggest? >I took your data, and reformated a little bit. Percentages at the end of line is difference between Dom0 and DomU. It seems to me that reads aren''t too bad as long as you keep a little bit of asynchronicity. Writes are worse, which isn''t entirely surprising. Random operations are much better, presumably because the overhead is now more hidden behind the much larger overhead of the disk-seek operation. The only way to avoid having SOME overhead between the DomU and Dom0 file-access would be to have a dedicated IO-device for each DomU, which becomes a bit unpractical if you have many DomU''s that are all IO-dependant. But if you have for example one DomU that is a data-base server, that may be the right solution. Note also that just because it uses threads doesn''t necessarily mean that the IO requests are saturating the bus, so the latency added by the DomU to Dom0 may have more to do with the overall performance, which in an application that actually does some REAL work between each IO request will be somewhat less (if it''s using some suitable prefetching operations, of course). Sequential Reads Num Dom0 DomU Thr Rate Rate --- ------ ------ 1 59.41 47.22 20.52% 2 55.44 50.51 8.89% 4 54.86 34.17 37.71% 8 45.94 30.68 33.22% 8.89% Random Reads Num Dom0 DomU Thr Rate Rate --- ------ ------ 1 0.94 1.39 -47.87% 2 1.61 1.42 11.80% 4 2.29 2.86 -24.89% 8 2.72 4.08 -50.00% -50.00% Sequential Writes Num Dom0 DomU Thr Rate Rate --- ------ ------ 1 10.48 6.98 33.40% 2 10.08 6 40.48% 4 9.74 5.88 39.63% 8 9.56 5.27 44.87% 33.40% Random Writes Num Dom0 DomU Thr Rate Rate --- ------ ------ 1 1.23 1.01 17.89% 2 1.24 1.03 16.94% 4 1.22 0.98 19.67% 8 1.12 0.93 16.96% 16.94% -- Mats _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users