I'm setting up a new fileserver. It has two RAID controllers, a PERC 3/DI
providing mirrored system disks and a PERC 3/DC providing a 1TB RAID5 volume
consisting of eight 144GB U160 drives. This will serve NFS, Samba and sftp
clients for about 200 users.
The logical drive was created with the following settings:
RAID = 5
stripe size = 32kb
write policy = wrback
read policy = adaptive
cache policy = CachedIO
#stripes = 8
I intend to revisit the effects of stripe size and read/write/cache policies
at a later stage, but any advice on selecting the appropriate parameters is
greatly appreciated.
I considered other file systems but have chosen ext3 for various reasons not
discussed here.
When creating the file system I used the following options:
-O sparse_super -O dir_index -R stride=8
-O sparse_super because I don't want to waste too much space
-O dir_index because this is supposed to help file listing performance with
Samba shares -R stride=8 because of the 32kb stripe size (4kb*8=32kb)
The next stage in my plan is to optimise file system performance. I've
decided to use bonnie++ (local execution) for the initial tests. I will
then extend the tests to factor in network file system performance (via NFS
and Samba) with multiple simultaneous loads.
I decided to investigate the use of an external log device. I have read
that it can improve performance for RAID 5 volumes, especially when the log
file is on a separate mirrored volume. I have also read that using
data=journal can further improve performance for applications such as NFS.
For these tests I created a 256MB partition in the mirrored disk set for the
log device: /dev/sdb6. The RAID5 volume is a 500GB partition: /dev/sda1.
Here is a summary of the results:
Log location:
- log location does not affect the file system performance as measured by
the sequential output and sequential input tests
- random seek rate improves with an external log (25%)
- log location affects the random create "create" test results and has
a
massive impact on the random create "read" and "delete" test
results
Data journaling:
- data journaling does not affect sequential input or random seek test
results
- data journaling significantly reduces the performance of the file system
with regard to sequential output, sequential create and random create
(between 25% and 56% decrease).
- data journaling improves file system performance in only one area, the
random create "delete" rate.
Writeback journaling:
- data=writeback has few performance benefits over data=ordered:
- sequential create "read" (50% faster)
- random create "delete" (100% faster)
- data=writeback under performed for the random create "create" and
"read"
tests
Please provide feedback on these results, especially on the significance of
the random create tests.
I would also like to hear any suggestions on how I might further improve
file system performance or my test methodology.
The tests...
# internal log
mkfs.ext3 -v -O sparse_super -O dir_index -R stride=8 /dev/sda1 mount -t
ext3 /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
ss1 4G 28235 94 38522 28 17222 8 22257 61 40274 10 598.0
1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
128:20000:16/512 1566 17 11216 24 9255 43 1872 20 500 1 101
1
ss1,4G,28235,94,38522,28,17222,8,22257,61,40274,10,598.0,1,128:20000:16/512,
1566,17,11216,24,9255,43,1872,20,500,1,101,1
# external log
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
ss1 4G 28557 95 38721 28 17354 8 22077 60 40036 10 742.7
1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
128:20000:16/512 1558 17 24247 51 20063 92 2512 28 39603 100 8696
48
ss1,4G,28557,95,38721,28,17354,8,22077,60,40036,10,742.7,1,128:20000:16/512,
1558,17,24247,51,20063,92,2512,28,39603,100,8696,48
# external data log
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 -o data=journal /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
ss1 4G 12369 47 17023 21 12894 14 22120 61 40720 10 736.0
2
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
128:20000:16/512 1128 16 5103 12 20367 95 1144 16 725 2 17172
94
ss1,4G,12369,47,17023,21,12894,14,22120,61,40720,10,736.0,2,128:20000:16/512
,1128,16,5103,12,20367,95,1144,16,725,2,17172,94
# external data log with writeback
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 -o data=writeback /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
ss1 4G 28205 95 37493 25 17134 7 21320 59 40395 10 756.3
2
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
128:20000:16/512 1637 17 36370 76 20284 94 2109 22 27351 70 17057
94
ss1,4G,28205,95,37493,25,17134,7,21320,59,40395,10,756.3,2,128:20000:16/512,
1637,17,36370,76,20284,94,2109,22,27351,70,17057,94
--
Simon Oliver
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ext3.pdf
Type: application/pdf
Size: 16811 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/ext3-users/attachments/20040714/e773bf4a/attachment.pdf>