Hello, I have a production mail server with maildir++ structure and about 250GB (~10 millions) of files on the ext3 partition on RAID5. It's mounted with noatime option. These mail server is responsible to local delivery and storing mail messages. System has Debian Squeeze installed and Exim as MDA + Dovecot as IMAP+POP3 server. Bonnie results are terrible. Sequential output for Block and Rewrite are 10722ms and 9232ms. So if there is a 1000 messages in the mail queue load is extremely high, delivery time is very big and server can hang. I did not see such problems with UFS on FreeBSD server. As I understand ext3 file system is really bad for such configurations with Maildir++ (many smaill files)? Is there a way to decrease disk latency on ext3 or speed up it? With regards, Andrey
Thank you for reply, BTW, other webserver has almost the same bonnie results (10283ms and 5884ms) on ext3 partition with 45GB of data (1.5 millions of files)?! Hardware and RAID5(also hardware) are the same: HP Proliant DL380 G4 with SmartArray 6i controller (as I see it comes with 128MB BBWC enabler but not kit). I did not tried to mount fs with barriers disabled. Does it have any crititcal risks? Bonnie tests was performed in the morning when we have a mininmal user load. But why the same server with the same RAID(4 disks) but with FreeBSD+UFS was much better? I guess problem is in ext3 then? With regards, Andrey. 23.09.2011 11:31, Janne Pikkarainen ?????:> Hello, > > On 09/23/2011 08:51 AM, Andrey wrote: >> Hello, >> >> I have a production mail server with maildir++ structure and about >> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's >> mounted with noatime option. These mail server is responsible to local >> delivery and storing mail messages. >> >> System has Debian Squeeze installed and Exim as MDA + Dovecot as >> IMAP+POP3 server. >> >> Bonnie results are terrible. Sequential output for Block and Rewrite >> are 10722ms and 9232ms. So if there is a 1000 messages in the mail >> queue load is extremely high, delivery time is very big and server can >> hang. I did not see such problems with UFS on FreeBSD server. >> >> As I understand ext3 file system is really bad for such configurations >> with Maildir++ (many smaill files)? Is there a way to decrease disk >> latency on ext3 or speed up it? >> >> With regards, Andrey >> >> ___ > > (replying off-list, so the ext3 developers will not start a flamewar) > > In my opinion ext3 is a terrible file system for your kind of workload, > especially if you have lots of concurrent clients accessing their > mailboxes. Even though ext3 has evolved over the years and has gained > features such as directory indexes, it still is not good for tens of > million of frequently changing small files with lots of concurrency. > Been there, done that, not gonna do it again. I administer servers with > 50 000 - 100 000 user accounts, with couple of thousands active IMAP > connections. > > Personally I switched from ext3 to ReiserFS many years ago and happily > used it between 2004-2008, then after things went downhill from ReiserFS > development point of view, I switched to XFS during a server hardware > refresh. ReiserFS was excellent, but it really started to slow down if > file system was more than 85% full and it also got fragmented over time. > > XFS has been rock-solid and fast since 2008 for me, but it has an > achilles heel of its own: if I need to remove lots of files from a huge > directory tree, the delete performance is quite sucky compared to other > file systems. This has been improved in the later kernel versions with > the new delaylog parameter, but how much, I've not yet tested. > > All this said, the performance of ext3 should not be THAT bad you are > describing. Is the bonnie result done while the server is idle or while > it has mail clients accessing it all the time? If you have hardware > RAID, is there a battery-backed up write cache and are you sure it's > enabled? Also, have you tried to mount your file system with barriers > disabled? What kind of server setup you have? > > Something is very wrong. > > Best regards, > > Janne Pikkarainen > >
On 09/22/2011 11:51 PM, Andrey wrote:> Hello, > > I have a production mail server with maildir++ structure and about > 250GB (~10 millions) of files on the ext3 partition on RAID5. It's > mounted with noatime option. These mail server is responsible to local > delivery and storing mail messages. > > System has Debian Squeeze installed and Exim as MDA + Dovecot as > IMAP+POP3 server. > > Bonnie results are terrible. Sequential output for Block and Rewrite > are 10722ms and 9232ms. So if there is a 1000 messages in the mail > queue load is extremely high, delivery time is very big and server can > hang. I did not see such problems with UFS on FreeBSD server. > > As I understand ext3 file system is really bad for such configurations > with Maildir++ (many smaill files)? Is there a way to decrease disk > latency on ext3 or speed up it? >My guess is that your problem is many files in one directory not necessarily having many files on the whole file system. In my experience large directories eat ext3's lunch. The introduction of indexing did help but it still fell behind on performance when compared to some other file systems. You may want to make sure your file system has indexing turned on but with the vintage of your Debian I would assume it is on by default. I ran into this problem many years ago (before indexing was an ext3 option). It was even worse as the Maildir storage was being accessed over NFS. Ended up eventually biting the bullet and moving to WAFL (NetApp). My guess is that users trying to access these large directories via IMAP and POP are also facing large delays and possibly even time outs. Steven
Let me to share some testing RAID 5 results with bonnie++:
ext3 (defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 242 96 22458 10 8826 2 1854 98 120985 11
317.1 3
Latency 211ms 896ms 720ms 22258us 18733us
622ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++
15404 35
Latency 12284us 992us 1029us 432us 140us
76us
ext3 (-T small,defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 229 98 4989 5 3862 1 1762 97 91111 9
266.6 6
Latency 79046us 22858ms 2577ms 19253us 12120us
767ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++
10347 26
Latency 9968us 977us 964us 482us 144us
178us
ext3 (-T news,defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 237 95 22807 11 8807 2 1897 99 121893 11
324.6 5
Latency 223ms 808ms 523ms 13765us 11049us
831ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++
15460 35
Latency 417us 984us 1024us 430us 140us
175us
ext4 (defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 256 98 21495 6 9896 2 1771 99 125775 11
349.7 5
Latency 37738us 992ms 3490ms 10811us 12045us
495ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++
17428 45
Latency 424us 982us 1023us 367us 139us
174us
xfs(defaults,noatime,logbufs=8,logbsize=131072):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 476 96 35129 9 12524 3 1417 99 124716 12
445.9 9
Latency 19798us 420ms 721ms 14122us 9394us
131ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++
1346 8
Latency 104ms 291us 48604us 109ms 45us
227ms
It seems that latency is big in whole results, best is for XFS. It is
tempting me to think that there are some RAID 5 issues here. It's really
strange that block writing for SCSI server disks in RAID5 is no more
than 30MB/sec(XFS). I guess I should consider XFS file system or
different RAID configuration. May be someone can comment this strange
benchmark result? Will very appreciate that.
With regards, Andrey.
23.09.2011 11:31, Janne Pikkarainen ?????:> Hello,
>
> On 09/23/2011 08:51 AM, Andrey wrote:
>> Hello,
>>
>> I have a production mail server with maildir++ structure and about
>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>> mounted with noatime option. These mail server is responsible to local
>> delivery and storing mail messages.
>>
>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>> IMAP+POP3 server.
>>
>> Bonnie results are terrible. Sequential output for Block and Rewrite
>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>> queue load is extremely high, delivery time is very big and server can
>> hang. I did not see such problems with UFS on FreeBSD server.
>>
>> As I understand ext3 file system is really bad for such configurations
>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>> latency on ext3 or speed up it?
>>
>> With regards, Andrey
>>
>> ___
>