zfs discuss - Jan 2007 - Help understanding some benchmark results

G''day, all,

So, I''ve decided to migrate my home server from Linux+swRAID+LVM to
Solaris+ZFS, because it seems to hold much better promise for data integrity,
which is my primary concern.

However, naturally, I decided to do some benchmarks in the process, and I
don''t understand why the results are what they are.  I though I had a
reasonable understanding of ZFS, but now I''m not so sure.

I''ve used bonnie++ and a variety of Linux RAID configs below to
approximate equivalent ZFS configurations and compare.  I do realise
they''re not exactly the same thing, but it seems to me they''re
reasonable comparisons and should return at least somewhat similar performance. 
I also realise bonnie++ is not an especially comprehensive or complex benchmark,
but ultimately I don''t really care about performance and this was only
done out of curiosity.


The executive summary is that ZFS write performance appears to be relatively
awful (all the time), and it''s read performance is relatively good most
of the time (with "striping", "mirroring" and
raidz[2]''s with fewer numbers of disks).


Examples:
* 8-disk RAID0 on Linux returns about 190MB/s write and 245MB/sec read, while a
ZFS raidz using the same disks returns about 120MB/sec write, but 420MB/sec
read.
* 16-disk RAID10 on Linux returns 165MB/sec and 440MB/sec write and read, while
a ZFS pool with 8 mirrored disks returns 140MB/sec write and 410MB/sec read.
* 16-disk RAID6 on Linux returns 126MB/sec write, 162MB/sec read, while a
16-disk raidz2 returns 80MB/sec write and 142MB/sec read.

The biggest problem I am having understanding "why is it so", is
because I was under the impression with ZFS''s CoW, etc, that writing
(*especially* writes like this, to a raidz array) should be much faster than a
regular old-fashioned RAID6 array.

I certainly can''t complain about the read speed, however - 400-odd
MB/sec out of this old beastie is pretty impressive :).

Help ?  Have I missed something obvious or done something silly ?
(Additionally, from the Linux perspective, why are reads so slow ?)


PS: Hardware specs are
Dual 2.8Ghz Xeon, 533Mhz bus, 2G RAM.
Two SuperMicro 8-port SATA controllers - AOC-SAT2-MV8 - on separate PCI-X buses
(Marvell  88SX6081 based,
http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm)
16 Seagate 250G drives, 8M cache, ST3250820AS
Motherboard is a pull from an old Gateway dual Xeon server.


PPS: Benchmark data (this will look awful with a proportional font):
(With the Linux "RAID50" and "RAID60", I created two RAID5
and RAID6 arrays, respectively, then create a RAID0 array over the top of them. 
For "RAID10" I used the ''builtin'' RAID10.)

Single drive
bonnie++ -d /mnt/ -s 8g -u0 -f
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           42807  21 25278   7           65191   6 118.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,42807,21,25278,7,,,65191,6,118.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal c0t0d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           46916  18 27814  11           62103   8 212.5   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 15522  99 +++++ +++ 24489  99 10140  99 +++++ +++ 5196  99
nitrogen,8G,,,46916,18,27814,11,,,62103,8,212.5,0,16,15522,99,+++++,+++,24489,99,10140,99,+++++,+++,5196,99



8 disk RAID0, one controller, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           190853  95 119364  42           245770  28 421.4  
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,190853,95,119364,42,,,245770,28,421.4,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal c0t0d0
zpool add internal c0t1d0
zpool add internal c0t2d0
zpool add internal c0t3d0
zpool add internal c0t4d0
zpool add internal c0t5d0
zpool add internal c0t6d0
zpool add internal c0t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           121711  47 100927  43           424900  75 617.4  
3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 14719  99 +++++ +++ 24421  99 14013  99 +++++ +++ 25612  99
nitrogen,8G,,,121711,47,100927,43,,,424900,75,617.4,3,16,14719,99,+++++,+++,24421,99,14013,99,+++++,+++,25612,99



16 disk RAID0, two controllers, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           189351  95 167948  65           442343  52 461.2  
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,189351,95,167948,65,,,442343,52,461.2,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal c0t0d0
zpool add internal c0t1d0
zpool add internal c0t2d0
zpool add internal c0t3d0
zpool add internal c0t4d0
zpool add internal c0t5d0
zpool add internal c0t6d0
zpool add internal c0t7d0
zpool add internal c1t0d0
zpool add internal c1t1d0
zpool add internal c1t2d0
zpool add internal c1t3d0
zpool add internal c1t4d0
zpool add internal c1t5d0
zpool add internal c1t6d0
zpool add internal c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           130142  50 133316  57           412643  68 681.2  
3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 15778  99 +++++ +++ 24652  99  5835  99 +++++ +++ 22881  99
nitrogen,8G,,,130142,50,133316,57,,,412643,68,681.2,3,16,15778,99,+++++,+++,24652,99,5835,99,+++++,+++,22881,99



16 disk RAID10, two controllers, 128k stripe

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           165585  93 107365  45           268066  34 420.7  
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,165585,93,107365,45,,,268066,34,420.7,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal mirror c0t0d0 c1t0d0
zpool add internal mirror c0t1d0 c1t1d0
zpool add internal mirror c0t2d0 c1t2d0
zpool add internal mirror c0t3d0 c1t3d0
zpool add internal mirror c0t4d0 c1t4d0
zpool add internal mirror c0t5d0 c1t5d0
zpool add internal mirror c0t6d0 c1t6d0
zpool add internal mirror c0t7d0 c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           139750  54 120402  52           415646  73 702.6  
3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 14652  99 +++++ +++ 24324  99 14459  99 +++++ +++ 24347  99
nitrogen,8G,,,139750,54,120402,52,,,415646,73,702.6,3,16,14652,99,+++++,+++,24324,99,14459,99,+++++,+++,24347,99



8 disk RAID5, one controller, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           107288  67 54156  38           157472  47 299.2  
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,107288,67,54156,38,,,157472,47,299.2,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal raidz1 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0
c0t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           80800  31 57461  28           216634  43 188.6   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 15646  99 +++++ +++ 18737  99 13388  99 +++++ +++ 25170  99
nitrogen,8G,,,80800,31,57461,28,,,216634,43,188.6,1,16,15646,99,+++++,+++,18737,99,13388,99,+++++,+++,25170,99



16 disk RAID50, two controllers, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           134082  89 52457  32           105118  30 456.3  
2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,134082,89,52457,32,,,105118,30,456.3,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool add internal raidz1 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0
c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           82959  32 51422  33           193962  62 233.8   4
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 16458  99 +++++ +++ 16412  99 13135  99 +++++ +++ 11576  99
nitrogen,8G,,,82959,32,51422,33,,,193962,62,233.8,4,16,16458,99,+++++,+++,16412,99,13135,99,+++++,+++,11576,99


8 disk RAID6, one controller, 128k stripe

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           96770  61 51418  34           148072  44 305.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,96770,61,51418,34,,,148072,44,305.9,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal raidz2 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0
c0t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           89015  34 58932  28           188166  36 178.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 15475  99 +++++ +++ 22184  99  9085  99 +++++ +++ 24124  99
nitrogen,8G,,,89015,34,58932,28,,,188166,36,178.3,1,16,15475,99,+++++,+++,22184,99,9085,99,+++++,+++,24124,99



16 disk RAID60, two controllers, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           131519  88 51273  31           102324  30 419.2  
2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,131519,88,51273,31,,,102324,30,419.2,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool add internal raidz2 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0
c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           86688  33 46889  33           170449  58 254.2   5
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 16009  99 +++++ +++ 24150  99 12990  99 +++++ +++ 22813  99
nitrogen,8G,,,86688,33,46889,33,,,170449,58,254.2,5,16,16009,99,+++++,+++,24150,99,12990,99,+++++,+++,22813,99



16 disk RAID5, two controllers, 128k stripe

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           135052  89 71118  56           167523  69 412.2  
2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,135052,89,71118,56,,,167523,69,412.2,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal raidz1 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0
c0t7d0 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           90557  34 44982  37           147654  69 209.4   8
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 15631  99 +++++ +++ 21228  99 10132  99 +++++ +++ 24391  99
nitrogen,8G,,,90557,34,44982,37,,,147654,69,209.4,8,16,15631,99,+++++,+++,21228,99,10132,99,+++++,+++,24391,99



16 disk RAID6, two controllers, 128k stripe
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G           126421  84 68968  55           162105  65 360.0  
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
csmith-desktop,8G,,,126421,84,68968,55,,,162105,65,360.0,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

zpool create internal raidz2 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0
c0t7d0 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen         8G           80457  31 38875  35           142049  64 193.1   7
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16 14426  99 +++++ +++ 21855  99 10068  99 +++++ +++ 24235  99
nitrogen,8G,,,80457,31,38875,35,,,142049,64,193.1,7,16,14426,99,+++++,+++,21855,99,10068,99,+++++,+++,24235,99
 
 
This message posted from opensolaris.org

Chris Smith wrote:
> G''day, all,
> 
> So, I''ve decided to migrate my home server from Linux+swRAID+LVM
to Solaris+ZFS, because it seems to hold much better promise for data integrity,
which is my primary concern.
> 
> However, naturally, I decided to do some benchmarks in the process, and I
don''t understand why the results are what they are.  I though I had a
reasonable understanding of ZFS, but now I''m not so sure.
> 
> I''ve used bonnie++ and a variety of Linux RAID configs below to
approximate equivalent ZFS configurations and compare.  I do realise
they''re not exactly the same thing, but it seems to me they''re
reasonable comparisons and should return at least somewhat similar performance. 
I also realise bonnie++ is not an especially comprehensive or complex benchmark,
but ultimately I don''t really care about performance and this was only
done out of curiosity.
> 
> 
> The executive summary is that ZFS write performance appears to be
relatively awful (all the time), and it''s read performance is
relatively good most of the time (with "striping",
"mirroring" and raidz[2]''s with fewer numbers of disks).
> 
> 
> Examples:
> * 8-disk RAID0 on Linux returns about 190MB/s write and 245MB/sec read,
while a ZFS raidz using the same disks returns about 120MB/sec write, but
420MB/sec read.
> * 16-disk RAID10 on Linux returns 165MB/sec and 440MB/sec write and read,
while a ZFS pool with 8 mirrored disks returns 140MB/sec write and 410MB/sec
read.
> * 16-disk RAID6 on Linux returns 126MB/sec write, 162MB/sec read, while a
16-disk raidz2 returns 80MB/sec write and 142MB/sec read.
> 
> The biggest problem I am having understanding "why is it so", is
because I was under the impression with ZFS''s CoW, etc, that writing
(*especially* writes like this, to a raidz array) should be much faster than a
regular old-fashioned RAID6 array.
> 
> I certainly can''t complain about the read speed, however - 400-odd
MB/sec out of this old beastie is pretty impressive :).
> 
> Help ?  Have I missed something obvious or done something silly ?
> (Additionally, from the Linux perspective, why are reads so slow ?)
> 
> 
What build/version of Solaris/ZFS are you using?

What block size are you using for writes in bonnie++?  I
find performance on streaming writes is better w/ larger writes.

What happens when you run two threads at once?  Does write performance
improve?

Does zpool iostat -v 1 report anything interesting during the benchmark?
What about iostat -x 1?  Is one disk significantly more busy than the
others?

I have a 4x 500GB disk raidz config w/ a 2.6 GHz dual core at home;
this config sustains approx 120 MB/sec on reads and writes on single
or multiple streams.  I''m running build 55; the box has a SI controller
running in PATA compat. mode.

One of the challenging aspects of performance work on these sorts of
things is separating out drivers vs cpus vs memory bandwidth vs disk
behavior vs intrinsic FS behavior.


- Bart



-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

> 
> What build/version of Solaris/ZFS are you using?
Solaris 11/06.
bash-3.00# uname -a
SunOS nitrogen 5.10 Generic_118855-33 i86pc i386 i86pc
bash-3.00#
> What block size are you using for writes in bonnie++?
>  I
> ind performance on streaming writes is better w/
> larger writes.
I''m afraid I don''t know what block size bonnie++ uses by
default - I''m not specifying one on the commandline.
> What happens when you run two threads at once?  Does
> write performance
> improve?
If I use raidz, no (overall throughput is actually nearly halved !).  If I use
"RAID0" (just striped disks, no redundancy) it improves (significantly
in some cases).
> Does zpool iostat -v 1 report anything interesting
> during the benchmark?
> What about iostat -x 1?  Is one disk significantly
> more busy than the
> others?
Nothing looked what I would consider "interesting".  The load seemed
quite evenly balanced across all the disks (based on my Mk 1 eyeball). 
I''ve pasted some of it below, from halfway through the write cycle of
each run:

(The configuration in this particular run is two 8-disk raidzs.  The actual
configuration I''ll probably end up using a 16-disk raidz2, to maximise
my disk space/reliability ratio.)

Single run:
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen         8G           91101  35 76667  42           216524  49 220.0   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 14261  99 +++++ +++ 24105  99 14395  99 +++++ +++ 25543  99
nitrogen,8G,,,91101,35,76667,42,,,216524,49,220.0,2,16,14261,99,+++++,+++,24105,99,14395,99,+++++,+++,25543,99

zpool iostat -v 1
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
internal    4.31G  3.63T      0    838      0  89.0M
  raidz1    2.14G  1.82T      0    423      0  45.1M
    c0t0d0      -      -      0    397      0  6.48M
    c0t1d0      -      -      0    396      0  6.43M
    c0t2d0      -      -      0    396      0  6.47M
    c0t3d0      -      -      0    386      0  6.43M
    c0t4d0      -      -      0    398      0  6.48M
    c0t5d0      -      -      0    397      0  6.43M
    c0t6d0      -      -      0    396      0  6.47M
    c0t7d0      -      -      0    386      0  6.43M
  raidz1    2.17G  1.82T      0    414      0  43.9M
    c1t0d0      -      -      0    392      0  6.32M
    c1t1d0      -      -      0    391      0  6.28M
    c1t2d0      -      -      0    391      0  6.32M
    c1t3d0      -      -      0    381      0  6.27M
    c1t4d0      -      -      0    392      0  6.32M
    c1t5d0      -      -      0    390      0  6.28M
    c1t6d0      -      -      0    386      0  6.32M
    c1t7d0      -      -      0    381      0  6.27M
----------  -----  -----  -----  -----  -----  -----

iostat -x 1
                  extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
cmdk0        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
fd0          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd0          0.0  372.0    0.0 5691.5  0.1  0.1    0.6   3  12
sd1          0.0  372.0    0.0 5653.1  0.0  0.1    0.2   0   8
sd2          0.0  374.0    0.0 5692.5  0.1  0.1    0.5   3  12
sd3          0.0  353.1    0.0 5641.6  0.1  0.1    0.6   4  12
sd4          0.0  380.0    0.0 5690.4  0.0  0.1    0.3   1   9
sd5          0.0  380.0    0.0 5652.0  0.0  0.1    0.2   0   9
sd6          0.0  377.0    0.0 5695.4  0.0  0.1    0.3   1   9
sd7          0.0  360.0    0.0 5644.5  0.1  0.1    0.5   2  11
sd8          0.0  381.0    0.0 5920.3  0.0  0.1    0.3   1  10
sd9          0.0  374.0    0.0 5879.9  0.1  0.1    0.5   3  12
sd10         0.0  382.0    0.0 5937.7  0.0  0.1    0.3   1  10
sd11         0.0  356.1    0.0 5888.8  0.1  0.1    0.5   3  12
sd12         0.0  377.0    0.0 5938.2  0.0  0.1    0.3   0   9
sd13         0.0  370.0    0.0 5892.8  0.1  0.1    0.6   4  12
sd14         0.0  376.0    0.0 5932.2  0.0  0.1    0.3   1  10
sd15         0.0  360.0    0.0 5884.3  0.1  0.1    0.5   3  11
sd16         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
st4          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
nfs1         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0


Two simultaneous runs:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen         8G           31575  19 15822  19           87277  41 128.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 14616  99 +++++ +++ 21683  99  4141  99 +++++ +++  5630  98
nitrogen,8G,,,31575,19,15822,19,,,87277,41,128.8,1,16,14616,99,+++++,+++,21683,99,4141,99,+++++,+++,5630,98

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen         8G           29943  18 16116  20           90079  41 125.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3257  96 +++++ +++  2542  80 17372  99 +++++ +++ 24678  99
nitrogen,8G,,,29943,18,16116,20,,,90079,41,125.3,1,16,3257,96,+++++,+++,2542,80,17372,99,+++++,+++,24678,99

zpool iostat -v 1
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
internal    9.93G  3.63T      0    810      0  75.5M
  raidz1    4.97G  1.81T      0    412      0  37.8M
    c0t0d0      -      -      0    372      0  5.44M
    c0t1d0      -      -      0    370      0  5.40M
    c0t2d0      -      -      0    372      0  5.44M
    c0t3d0      -      -      0    315      0  5.39M
    c0t4d0      -      -      0    370      0  5.44M
    c0t5d0      -      -      0    370      0  5.40M
    c0t6d0      -      -      0    372      0  5.44M
    c0t7d0      -      -      0    349      0  5.39M
  raidz1    4.96G  1.81T      0    397      0  37.8M
    c1t0d0      -      -      0    366      0  5.44M
    c1t1d0      -      -      0    295      0  5.40M
    c1t2d0      -      -      0    369      0  5.44M
    c1t3d0      -      -      0    290      0  5.39M
    c1t4d0      -      -      0    360      0  5.43M
    c1t5d0      -      -      0    300      0  5.39M
    c1t6d0      -      -      0    360      0  5.43M
    c1t7d0      -      -      0    308      0  5.38M
----------  -----  -----  -----  -----  -----  -----

iostat -x 1
                  extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
cmdk0        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
fd0          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd0          0.0  384.5    0.0 5635.2  0.0  0.1    0.4   2  11
sd1          0.0  387.5    0.0 5599.1  0.0  0.3    0.8   1  28
sd2          0.0  384.5    0.0 5634.2  0.1  0.1    0.5   4  13
sd3          0.0  322.4    0.0 5581.1  0.2  0.2    1.2   9  15
sd4          0.0  380.5    0.0 5633.2  0.0  0.1    0.2   0   9
sd5          0.0  380.5    0.0 5591.2  0.0  0.1    0.3   1   9
sd6          0.0  382.5    0.0 5630.7  0.0  0.1    0.3   1   9
sd7          0.0  359.5    0.0 5580.1  0.0  0.1    0.3   0   9
sd8          0.0  363.5    0.0 5606.7  0.0  0.1    0.3   1   9
sd9          0.0  291.4    0.0 5567.6  0.3  0.2    1.8  11  18
sd10         0.0  362.5    0.0 5605.7  0.1  0.1    0.7   4  14
sd11         0.0  285.4    0.0 5555.6  0.2  0.2    1.3   7  16
sd12         0.0  358.5    0.0 5603.7  0.0  0.1    0.2   0   8
sd13         0.0  299.4    0.0 5563.1  0.2  0.2    1.4   8  18
sd14         0.0  357.5    0.0 5602.2  0.0  0.1    0.3   1   9
sd15         0.0  308.4    0.0 5555.6  0.2  0.2    1.2   6  18
sd16         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
st4          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
nfs1         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0

> I have a 4x 500GB disk raidz config w/ a 2.6 GHz dual
> core at home;
> this config sustains approx 120 MB/sec on reads and
> writes on single
> or multiple streams.  I''m running build 55; the box
> has a SI controller
> running in PATA compat. mode.
> 
> One of the challenging aspects of performance work on
> these sorts of
> things is separating out drivers vs cpus vs memory
> bandwidth vs disk
> behavior vs intrinsic FS behavior.
Some observations:
* This machine only has 32 bit CPUs.  Could that be limiting performance ?
* A single drive will hit ~60MB/s read and write.  Since these are only 7200rpm
SATA disks, that''s probably all they''ve got to give.
* A 4-disk (one controller) "RAID0" zpool delivers about 250MB/sec
reads, 95MB/sec writes.
* A 4-disk (two controllers) "RAID0" zpool delivers about 250MB/sec
reads, 95MB/sec writes.
* Two 4-disk (two controllers) "RAID0" zpools delivers about 350MB/sec
aggregate reads, 210MB/sec writes.
* Three 4-disk "RAID0" zpools delivers about 310MB/sec aggregate
reads, 170MB/sec writes.
* Four 4-disk "RAID0" zpools delivers... an incompleted benchmark. 
When the "rewrite" part of bonnie++ was running in this particular
test, frequently all IO to the zpools would stop, and subsequently all the
output from ''zpool iostat'' and ''iostat''
would freeze for lengthening periods of time.  Clearly some resource was
becoming exhausted.  Eventually I just ran out of patience and killed them all.
* One 8-disk "RAID0" zpool (single controller) delivers about
400MB/sec reads, 120MB/sec writes.
* One 8-disk "RAID0" zpool (four disks from each controller) delivers
about 400MB/sec reads, 140MB/sec writes.
* Two 8-disk "RAID0" zpools delivers about 550MB/sec aggregate on
reads, 220MB/sec writes.
* One 16-disk "RAID0" zpool will return 400MB/sec on reads, 135MB/sec
on writes.
* I would have thought the top-end performance to be a bit higher (or maybe
I''m being ambitious ?).  It may be hardware related - although (as long
as I''m reading the output of ''lspci -vt'' from the
Linux install right) each controller should be on its own 100Mhz, 64 bit PCI-X
bus.  This motherboard is several years old, however, so it''s possible
I''m just bumping into the limitations of its PCI implementation.  I
might try the cards in different slots when I get home tonight and see if that
makes a difference.
* Writes are a lot slower than reads across the board (I expected this, just
wondering if the results I''m getting are reasonable).
* Writes seem to be quite a bit slower relative to Linux s/w RAID as well.  Is
this just an engineering tradeoff ZFS has made to provide benefits in other
areas ?  What kind of write performance do people get out of those
honkin'' big x4500s ?
 
 
This message posted from opensolaris.org

Chris Smith wrote:
>> What build/version of Solaris/ZFS are you using?
> 
> Solaris 11/06.
>> What block size are you using for writes in bonnie++?
>>  I
>> ind performance on streaming writes is better w/
>> larger writes.
> 
> I''m afraid I don''t know what block size bonnie++ uses by
default - I''m not specifying one on the commandline.
> 
trussing bonnie will tell you... I just built bonnie++; by default it
uses 8k.
>> What happens when you run two threads at once?  Does
>> write performance
>> improve?
> 
> If I use raidz, no (overall throughput is actually nearly halved !).  If I
use "RAID0" (just striped disks, no redundancy) it improves
(significantly in some cases).
> 
Increasing the blocksize will help.  You can do that on bonnie++ like
this:

./bonnie++ -d /internal/ -s 8g:128k ...

Make sure you don''t have compression on....
> 
> Some observations:
> * This machine only has 32 bit CPUs.  Could that be limiting performance ?
It will, but it shouldn''t be too awful here.  You can lower kernelbase
to let the kernel have more of the RAM on the machine.  You''re more 
likely going to run into problems w/ the front side bus; my experience
w/ older Xeons is that one CPU could easily saturate the FSB; using the
other would just make things worse.  You should not be running into that
yet, either, though.  Offline one of the CPUs w/ psradm -f 1; reenable 
w/ psradm -n 1.
> * A single drive will hit ~60MB/s read and write.  Since these are only
7200rpm SATA disks, that''s probably all they''ve got to give.
On a good day on the right part of the drive... slowest to fastest 
sectors can be 2:1 in performance...

What can you get with your drives w/ dd to the raw device when not part 
of a pool?

Eg /bin/ptime dd if=/dev/zero of=/dev/dsk/... bs=128k bc=20000

You can also do this test to a file to see what the peak is going to be...
> What kind of write performance do people get out of those honkin''
big x4500s ?
>  
~2GB/sec locally, 1 GB/sec over the network.

This requires multiple writing threads; a single CPU just isn''t fast
enough to write 2GB/sec.

- Bart



-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

Unfortunately I''ve started restoring data onto my array (2.5TB @ the
20ish MB/sec my LTO2 drive maxes out at will take a while ;) ) so I
can''t do any more testing that involves destroying the zpool and/or
individual devices...

So all the numbers below are to a 16-disk raidz2 zpool (unless otherwise noted).
> > If I use raidz, no (overall throughput is actually
> nearly halved !).  If I use "RAID0" (just striped
> disks, no redundancy) it improves (significantly in
> some cases).
> > 
> 
> Increasing the blocksize will help.  You can do that
> on bonnie++ like
> this:
> 
> ./bonnie++ -d /internal/ -s 8g:128k ...
> 
> Make sure you don''t have compression on....
Didn''t seem to make any (significant) difference:

Single run:
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen    8G:128k           78974  10 46895  16           136737  30  89.1   4
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 12150  94 +++++ +++ 16762  97 11616  87 +++++ +++ 23418  99
nitrogen,8G:128k,,,78974,10,46895,16,,,136737,30,89.1,4,16,12150,94,+++++,+++,16762,97,11616,87,+++++,+++,23418,99

Two simultaneous runs:
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen    8G:128k           34556   9 13084  20           56964  21  44.0   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  9570  74 +++++ +++ 11259  85  7129  90 19005  95   605  74
nitrogen,8G:128k,,,34556,9,13084,20,,,56964,21,44.0,2,16,9570,74,+++++,+++,11259,85,7129,90,19005,95,605,74

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nitrogen    8G:128k           33355   8 12314  19           66295  21  47.1   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3365  48 +++++ +++  6118  74  1921  77 +++++ +++  1601  78
nitrogen,8G:128k,,,33355,8,12314,19,,,66295,21,47.1,2,16,3365,48,+++++,+++,6118,74,1921,77,+++++,+++,1601,78
> > Some observations:
> > * This machine only has 32 bit CPUs.  Could that be
> limiting performance ?
> 
> It will, but it shouldn''t be too awful here.  You can
> lower kernelbase
> to let the kernel have more of the RAM on the
> machine.  You''re more 
> likely going to run into problems w/ the front side
> bus; my experience
> w/ older Xeons is that one CPU could easily saturate
> the FSB; using the
> other would just make things worse.  You should not
> be running into that
> yet, either, though.  Offline one of the CPUs w/
> psradm -f 1; reenable 
> w/ psradm -n 1.
I tried this and it didn''t seem to make a difference.

According to vmstat, during the write phase of the benchmark, cpu idle% was
getting down around the 0 - 15% range and cpu sys% was 75 - 90%.  With two CPUs
active idle% and sys% were around 50% each.
> > * A single drive will hit ~60MB/s read and write.
> Since these are only 7200rpm SATA disks, that''s
>  probably all they''ve got to give.
> n a good day on the right part of the drive...
> slowest to fastest 
> sectors can be 2:1 in performance...
> 
> What can you get with your drives w/ dd to the raw
> device when not part 
> of a pool?
> 
> Eg /bin/ptime dd if=/dev/zero of=/dev/dsk/... bs=128k
> bc=20000
As I said I can''t write to the raw devices any more, but I tried this
with reads (of=/dev/null) and the dd processes finished between 3:45 (quickest)
and 4:15 (longest).  So if my maths is right that''s 2.5GB written per
process * 16 processes = 40GB in 4:15 = about 155MB/sec (eyeballing iostat -x
output during the run gave about the same value).

Interesting that this is _substantially_ less than the 400MB/sec a 16-disk RAID0
managed to achieve (albeit about on par with my 16-disk raidz2)...

Added to that, watching the disk activity with iostat -x indicated the
individual drive throughputs to vary frequently and significantly throughout the
test.  One second a drive would be hitting 50MB/sec, the next it would be down
around 7.5MB/sec, the next 25MB/sec, etc.  All the disks exhibited this
behaviour.


I''m impressed the x4500s can hit that sort of speed - clearly then my
performance is being limited by the hardware and not the software ;).

Are there any tuning knobs I can play with for ZFS and/or the SATA controllers
and/or the disks ?

Cheers,
CS
 
 
This message posted from opensolaris.org