CentOS - Sep 2011 - CentOS 6.0 and 3ware 9650SE series RAID Performance

Hello,

Does anyone have experience using a 3ware 9650SE series raid controller on
CentOS 6.0?

I am getting very sporadic throughput with moderately sized files (0.5-2GB)
on ext3.  I have tried most of the mount time tuning options:

* noatime
* trying different journal types
* setting commit=120 - helped a little

Even after these optimizations it doesn't seem like the raid array is
working as it should.  After a few 1GB writes (dd from /dev/zero to the
raid) kjournald runs for an hour and later writes are really slow.  Not only
that, using tw_cli is very slow ... whereas tw_cli is superfast if kjournald
is not churning away.

So it goes something like this

* write one 1GB file (486 MB/s)
* writes another (223 MB/s)
* any writes before the 120s commit kicks in, is ~200MB/s
* commit kicks in and kjournald starts churning
* writes are all over the map - 6-85MB/s

Maybe this is just the way it is, but it did not seem to be the case this
same hardware was running Fedora (Core) 9 and I have a similar machine where
this does not seem to be the case (I can't experiment on it at the moment).

The one thing I do not know, since I did not create the RAID or ext3
filesystem is whether the stride and stripe-width were properly selected to
match the 64k chunk size of the raid array.  I don't know how to tell from
tune2fs ... output below ... its either not there or by another name.

Any help or suggestions are appreciated.

Austin

Useful Information
=============
============== /proc/mounts ====================
/dev/sda1 /tonga_raid ext3
rw,noatime,errors=continue,nouser_xattr,noacl,commit=120,barrier=1,data=ordered
0 0


=========  tw_cli /c2 show diag ====================
### Time Stamp:        12:34:18 01-Sep-2011
### Host Name:         tonga
### Host Architecture: x86_64 (64 bit)
### OS Version:        Linux 2.6.32-71.29.1.el6.x86_64
### Model:             9650SE-8LPML
### Serial #:          L326025A8221043
### Controller ID:     2
### CLI Version:       2.00.11.016
### API Version:       2.08.00.017
### Driver Version:    2.26.02.014RH
### Firmware Version:  FE9X 3.08.00.016
### BIOS Version:      BE9X 3.08.00.004
### Available Memory:  224MB

=========================================================================Diagnostic
Information on Controller //.../c2/...
--------------------------------------------------------------------------
Event Trigger and Log Information:
Triggered Event(s)     ctlreset (controller soft reset)
    fwassert (firmware assert)
    driveerr (drive error)
Diagnostic log save mode = -
Parameter table does not exist


========== tune2fs -l /dev/sda1 ================tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   /data
Last mounted on:          <not available>
Filesystem UUID:          3c5f6dbb-d5dc-4f85-bc70-9b761c89c86e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              427245568
Block count:              1708965879
Reserved block count:     85448293
Free blocks:              809305423
Free inodes:              426287552
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      616
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Tue Sep  9 09:57:44 2008
Last mount time:          Thu Sep  1 12:40:01 2011
Last write time:          Thu Sep  1 12:40:01 2011
Mount count:              11
Maximum mount count:      -1
Last checked:             Tue Aug 30 23:12:49 2011
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:          256
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      0b324311-93a9-4c23-bf15-40965792029b
Journal backup:           inode blocks
==========================================================

===================  tw_cli info c2  ===========================
Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
 AVrfy
------------------------------------------------------------------------------
u0    RAID-5    OK             -       -       64K     6519.19   OFF    OFF


Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     931.51 GB   1953525168    WD-WCASJ1631953

p1     OK               u0     931.51 GB   1953525168    WD-WCASJ1622428

p2     OK               u0     931.51 GB   1953525168    WD-WCASJ1639721

p3     OK               u0     931.51 GB   1953525168    WD-WCASJ1636054

p4     OK               u0     931.51 GB   1953525168    WD-WCASJ1621694

p5     OK               u0     931.51 GB   1953525168    WD-WCASJ1636292

p6     OK               u0     931.51 GB   1953525168    WD-WCASJ1637586

p7     OK               u0     931.51 GB   1953525168    WD-WCASJ1637516

==========================================================-------------- next
part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20110901/9a3b3fa3/attachment.html>

Em 01-09-2011 17:41, Austin Godber escreveu:
>
> ===================  tw_cli info c2  ===========================> 
> Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
>  AVrfy
>
------------------------------------------------------------------------------
> u0    RAID-5    OK             -       -       64K     6519.19   OFF  
>  OFF    
I have the same controller on Centos 5.

Did you try to active Cache on 3ware?

[17:56:04 root at backup ~]# lspci | grep 3ware
01:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
[17:56:11 root at backup ~]# tw_cli /c4 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache
AVrfy
------------------------------------------------------------------------------
u0    RAID-5    OK             -       -       64K     5587.9    RiW
ON


In the past I had a scenario where I started with Cache OFF and had a
poor performance, specially for write. After set Cache RiW I got better
performance.

I didn't test that controller on CentOS 6, but is good to know if there
is some problem, because my company sells equipment with that controller.


Best regards,

-- 
Marcelo Beckmann
Suporte Corporativo - suporte at webers.com.br
Webers Tecnologia - http://www.webers.com.br
Curitiba       (PR) (41) 3094-6600
Rio de Janeiro (RJ) (21) 4007-1207
S?o Paulo      (SP) (11) 4007-1207

On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
> Hello,
> 
> Does anyone have experience using a 3ware 9650SE series raid controller on
CentOS 6.0?
----
use RAID 10

Unless something has changed, RAID 5 is notoriously slow on the 3Ware
controllers. Whatever you do will only incrementally speed things up. If
performance is desired, RAID 5 is not the way to go.

Craig