Lustre discuss - Apr 2012 - tuning for performance

Hello list, 
I''m trying to tune my lustreFS so that it''ll perform better,
because I''m sure I have a lot of low-level stuff that needs to be
optimzed.
My (poor) choice was to have a pair of MDS backed by a (supposely fast
Equallogic) iSCSI array, I also have three OSSes serving one OST each.
Those OST are a set of 6 NL SAS drives on RAID10 behind a Perc6/i (those Dell
cards I''m sure a lot know about, this one has BBU & 256Mb RAM).
system for those OST are on a separate mirrored pair of small disks
lnet sits on DDR IB (mellanox memory-less cards). Lustre version is 1.8.7-wc1 

I want to use more sensible settings for the RAID volume : 
First of all I would change the chunk/stripe/whatever-name-you-like size for the
volume to try to align with 1Mb/# of drives.
If I had four drives RAID10, i would have chosen 512k, since I''ve 6,
I''m more leaning towards 256 (means 1.33 writes per disk for 1Mb
''block'' which would most probably perform better than the 5.33
writes per disk using the 64k default setting).

Does that makes sense to any of you, or is this pointless ? 
Also, given the Raid card, do you think that it will be better to really align
to 1Mb by using a 5-drive RAID5 (so sparing one drive) or a RAID6 volume ?
then I could use the 256k stripe size AND be aligned, but I''ll have to
pay the cost for parity-computation and write(s)...

suggestions are more than welcomed... 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120403/c9dfc7a3/attachment.html

On 2012-04-03, at 7:04 AM, Francois Chassaing wrote:
> Hello list,
> I''m trying to tune my lustreFS so that it''ll perform
better, because I''m sure I have a lot of low-level stuff that needs to
be optimzed.
> My (poor) choice was to have a pair of MDS backed by a (supposely fast
Equallogic) iSCSI array, I also have three OSSes serving one OST each.
> Those OST are a set of 6 NL SAS drives on RAID10 behind a Perc6/i (those
Dell cards I''m sure a lot know about, this one has BBU & 256Mb
RAM). system for those OST are on a separate mirrored pair of small disks
> lnet sits on DDR IB (mellanox memory-less cards). Lustre version is
1.8.7-wc1
> 
> I want to use more sensible settings for the RAID volume :
> First of all I would change the chunk/stripe/whatever-name-you-like size
for the volume to try to align with 1Mb/# of drives.
> If I had four drives RAID10, i would have chosen 512k, since I''ve
6, I''m more leaning towards 256 (means 1.33 writes per disk for 1Mb
''block'' which would most probably perform better than the 5.33
writes per disk using the 64k default setting).
What you really should be looking at is to do component-level analysis using the
Lustre iokit to test the performance of each part of your IO stack.  First the
RAID level, next the OST level, and finally the client performance.
> Does that makes sense to any of you, or is this pointless ?
> Also, given the Raid card, do you think that it will be better to really
align to 1Mb by using a 5-drive RAID5 (so sparing one drive) or a RAID6 volume ?
> then I could use the 256k stripe size AND be aligned, but I''ll
have to pay the cost for parity-computation and write(s)...
I suspect you will get better performance and reliability with RAID-6 4+2 than
with RAID-10 3+3, but this is easily tested with sgpdd-survey from the iokit.

Note also that most deployments have multiple OSTs per OSS, since 6 drives is
typically not enough to saturate a decent server, and you get more performance
with less overhead.  You could probably put 4 OSTs into each OSS reasonably.  At
least it is something to consider for the future if you expand your filesystem.


Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/