Ext3 users - Sep 2011 - Fwd: Re: ext3 with maildir++ = huge disk latency and high load

23.09.2011 11:31, Janne Pikkarainen ?????:
Thank you for reply,

BTW, other webserver has almost the same bonnie results (10283ms and
5884ms) on ext3 partition with 45GB of data (1.5 millions of files)?!

Hardware and RAID5(also hardware) are the same: HP Proliant DL380 G4
with SmartArray 6i controller (as I see it comes with 128MB BBWC enabler
but not kit).

I did not tried to mount fs with barriers disabled. Does it have any
crititcal risks?

Bonnie tests was performed in the morning when we have a mininmal user load.

With regards, Andrey.
> Hello,
>
> On 09/23/2011 08:51 AM, Andrey wrote:
>> Hello,
>>
>> I have a production mail server with maildir++ structure and about
>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>> mounted with noatime option. These mail server is responsible to local
>> delivery and storing mail messages.
>>
>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>> IMAP+POP3 server.
>>
>> Bonnie results are terrible. Sequential output for Block and Rewrite
>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>> queue load is extremely high, delivery time is very big and server can
>> hang. I did not see such problems with UFS on FreeBSD server.
>>
>> As I understand ext3 file system is really bad for such configurations
>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>> latency on ext3 or speed up it?
>>
>> With regards, Andrey
>>
>> ___
>
> (replying off-list, so the ext3 developers will not start a flamewar)
>
> In my opinion ext3 is a terrible file system for your kind of workload,
> especially if you have lots of concurrent clients accessing their
> mailboxes. Even though ext3 has evolved over the years and has gained
> features such as directory indexes, it still is not good for tens of
> million of frequently changing small files with lots of concurrency.
> Been there, done that, not gonna do it again. I administer servers with
> 50 000 - 100 000 user accounts, with couple of thousands active IMAP
> connections.
>
> Personally I switched from ext3 to ReiserFS many years ago and happily
> used it between 2004-2008, then after things went downhill from ReiserFS
> development point of view, I switched to XFS during a server hardware
> refresh. ReiserFS was excellent, but it really started to slow down if
> file system was more than 85% full and it also got fragmented over time.
>
> XFS has been rock-solid and fast since 2008 for me, but it has an
> achilles heel of its own: if I need to remove lots of files from a huge
> directory tree, the delete performance is quite sucky compared to other
> file systems. This has been improved in the later kernel versions with
> the new delaylog parameter, but how much, I've not yet tested.
>
> All this said, the performance of ext3 should not be THAT bad you are
> describing. Is the bonnie result done while the server is idle or while
> it has mail clients accessing it all the time? If you have hardware
> RAID, is there a battery-backed up write cache and are you sure it's
> enabled? Also, have you tried to mount your file system with barriers
> disabled? What kind of server setup you have?
>
> Something is very wrong.
>
> Best regards,
>
> Janne Pikkarainen
>
>