CentOS - Oct 2005 - Best mkfs.ext2 performance options on RAID5 in CentOS 4.2

I can't seem to get the read and write performance better than
approximately 40MB/s on an ext2 file system.  IMO, this is horrible
performance for a 6-drive, hardware RAID 5 array.  Please have a look at
what I'm doing and let me know if anybody has any suggestions on how to
improve the performance...

System specs:
-----------------
2 x 2.8GHz Xeons 
6GB RAM
1 3ware 9500S-12
2 x 6-drive, RAID 5 arrays with a stripe size of 256KB.  Each array is
2.3TB after formatting.
ioscheduler set to use the deadline scheduler.

mkfs.ext2 options used:
------------------------
mkfs.ext2 -b 4096 -L /d01 -m 1 -O sparse_super,dir_index -R stride=64 -T
largefile /dev/sda1

I'm using a stride size of 64 since the ext2 block size is 4KB and the
array stripe size is 256KB (256/4 = 64).

Output of using bonnie++:
---------------------------
$ /usr/local/bonnie++/sbin/bonnie++ -d /d01/test -r 6144 -m
anchor_ext2_4k_64s
Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
anchor_ext2_4k_ 12G 41654  96 41937  11 30537   8 40676  88 233015  27
426.6   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16  3369  83 +++++ +++ +++++ +++  4142 100 +++++ +++
11728 100


Cheers!
Sean

On Mon, 31 Oct 2005 at 4:10pm, Sean Staats wrote
> I can't seem to get the read and write performance better than
> approximately 40MB/s on an ext2 file system.  IMO, this is horrible
> performance for a 6-drive, hardware RAID 5 array.  Please have a look at
> what I'm doing and let me know if anybody has any suggestions on how to
> improve the performance...
>
> System specs:
> -----------------
> 2 x 2.8GHz Xeons
> 6GB RAM
> 1 3ware 9500S-12
> 2 x 6-drive, RAID 5 arrays with a stripe size of 256KB.  Each array is
> 2.3TB after formatting.
> ioscheduler set to use the deadline scheduler.
Make sure you are using the absolute latest firmware for the 3ware board 
(2.08.00.005).  There is a bad cache allocation bug in previous firmwares 
that kills performance with more than one unit (as you have).

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Sean Staats <sstaats at questia.com> wrote:
> I can't seem to get the read and write performance better
> than approximately 40MB/s on an ext2 file system.
> IMO, this is horrible performance for a 6-drive, hardware
> RAID 5 array.
> 2 x 2.8GHz Xeons 
> 6GB RAM
> 1 3ware 9500S-12
You should be getting much, much higher than 40MBps reads on
any 3Ware controller.  As far as writes, there were some bugs
in the early 9.2 firmware that should be cleared up.  You
shouldn't be seeing anything that slow on a 9500S.

It's not the filesystem.  It's probably the card/array
configuration.
> 2 x 6-drive, RAID 5 arrays
Again, the early 9.2 firmware had a performance bug with
multiple volumes.  What is your firmware?
> with a stripe size of 256KB. 
Any reason you went with a 256KiB stripe size?
I typically stick with the 32KiB default.

-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith at ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)

blockdev --setra 16384 /dev/sda

Sean Staats wrote:
> I can't seem to get the read and write performance better than
> approximately 40MB/s on an ext2 file system.  IMO, this is horrible
> performance for a 6-drive, hardware RAID 5 array.  Please have a look at
> what I'm doing and let me know if anybody has any suggestions on how to
> improve the performance...
> 
> System specs:
> -----------------
> 2 x 2.8GHz Xeons 
> 6GB RAM
> 1 3ware 9500S-12
> 2 x 6-drive, RAID 5 arrays with a stripe size of 256KB.  Each array is
> 2.3TB after formatting.
> ioscheduler set to use the deadline scheduler.
> 
> mkfs.ext2 options used:
> ------------------------
> mkfs.ext2 -b 4096 -L /d01 -m 1 -O sparse_super,dir_index -R stride=64 -T
> largefile /dev/sda1
> 
> I'm using a stride size of 64 since the ext2 block size is 4KB and the
> array stripe size is 256KB (256/4 = 64).
> 
> Output of using bonnie++:
> ---------------------------
> $ /usr/local/bonnie++/sbin/bonnie++ -d /d01/test -r 6144 -m
> anchor_ext2_4k_64s
> Version  1.03       ------Sequential Output------ --Sequential Input-
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
> /sec %CP
> anchor_ext2_4k_ 12G 41654  96 41937  11 30537   8 40676  88 233015  27
> 426.6   1
>                     ------Sequential Create------ --------Random
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
> /sec %CP
>                  16  3369  83 +++++ +++ +++++ +++  4142 100 +++++ +++
> 11728 100
> 
> 
> Cheers!
> Sean
> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615

Quoting Sean Staats <sstaats at questia.com>:
> I can't seem to get the read and write performance better than
> approximately 40MB/s on an ext2 file system.  IMO, this is horrible
> performance for a 6-drive, hardware RAID 5 array.  Please have a look at
> what I'm doing and let me know if anybody has any suggestions on how to
> improve the performance...
> Output of using bonnie++:
[snip]
> ---------------------------
> $ /usr/local/bonnie++/sbin/bonnie++ -d /d01/test -r 6144 -m
> anchor_ext2_4k_64s
> Version  1.03       ------Sequential Output------ --Sequential Input-
> --Random-
>                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> anchor_ext2_4k_ 12G 41654  96 41937  11 30537   8 40676  88 233015  27
> 426.6   1
Correct me if I'm wrong, but you got 233MB/s for reads (the block read 
test). Assuming your disks can do 50MB/s sustained transfer rate each, 
you are preatty
darn close to the theoretical maximum of (6 - 1) * 50MB/s = 250MB/s for 6 disk
RAID5.

RAID5 as such is bad choice for file systems that will have more than 
about 30%
of writes (out of total I/O).  If most of the I/O will be writes, and you care
about performance, you should use RAID-10.  Remember, writes to Dumb,
not-optimized RAID5 implementation is slower than writing to a single 
disk. This is generic RAID wisdom, nothing to do with any particular 
implementation. In the worst case scenario, the write operation on 
6-disk RAID5 volume involves
reading a data block from 5 drives, calculating XOR, and writing back 
one block
of data and one block of checksum.  Whichever way you do it, it ain't gonna
be
fast.

For large sequential writes, RAID5 implementations can do a lot of 
optimizations
(reducing the number of reads for each write operation).  But they 
still need to
generate and write that additional XOR checksum, so it is going to be slower
than reading from that same volume.

The random writes to RAID5 volumes are always going to be terribly slow since
RAID5 implementation can't optimize them very well.  If they are limited to
small areas of data, large battery backedup on-controller cache might help
(since the blocks needed to re-calculate XOR checksum might already be in the
cache, and the actuall writes can be delayed in hope there'll be enough 
data to
write in the future to reduce number of needed reads).  If they are spread all
over 1TB volume, you are screwed, no (reasonable) amount of cache is going to
save ya.

Back to reads, you got 40MB/s for per-chr reads and 233MB/s for block 
reads. The difference between this two cases is not 3ware related, at 
least not in
your case it seems.  The per-chr test is reading one byte at a time, and it is
influenced by three factors: how well the C library is optimized (and how good
it is in buffering), the CPU speed and the disk speed.  If you look at CPU
column, you'll see that your CPU was 88% busy during this test (probably
most
time spent in a bonnie's loop that executes 12 billion getc() and the C 
library
itself).  So no matter how fast your drives are, you'd max out in per-chr
read
test at maybe 45-50MB/s with the CPU you have in the box.  Setting larger read
ahead (as Joe suggested) might help to squeeze couple of MB/s more in 
benchmark
tests, but probably not really worth it in real world applications.


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.