Hi, I have > 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Thanks for any hints an tricks, Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. ?35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Gesch?ftsf?hrer: Peer Heinlein -- Sitz: Berlin
What is the setup on the EVA, FC or iSCSI? Sent from my BlackBerry? wireless device -----Original Message----- From: Peer Heinlein <p.heinlein at heinlein-support.de> Sender: dovecot-bounces at dovecot.org Date: Tue, 8 Nov 2011 14:50:25 To: <dovecot at dovecot.org> Subject: [Dovecot] Performance-Tuning Hi, I have > 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Thanks for any hints an tricks, Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. ?35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Gesch?ftsf?hrer: Peer Heinlein -- Sitz: Berlin
On 08.11.2011 14:50, Peer Heinlein wrote:> *) Is mdbox really faster? I'd like to have mdbox to have better > performance > in running my backup-processes. But does it bring some performance > boosts > to?Hi Peer, We have switched our mailbox storage format from maildir to mdbox! Maildir is a disaster. (too many small files) After the migration to mdbox the performance has improved significantly. Conclusion: mdbox is great and much better performance than maildir! I would also recommend ext4. Best regards, Morten
We are very happy with mdbox+zlib+ext4 + iSCSI SAN (HP Lefthand in
our setup)
If you have CPU to spare, consider using zlib with mdbox. You are
trading CPU power (cheap) to get fewer IOPS (IOPS count is expensive).
Mdbox has halved our backup windows (2,8 TB uncompressed mailboxes, 2 TB
compressed) and backup software is happier because there are few
(100.000+ files with mdbox) to backup instead of several millions (Maildir)
Regards
Javier> Hi,
>
> I have> 11 TB hard used Mailstorage, saved als maildir in ext3 on HP
EVA.
>
> I always wanted to make some mesurements about several influences to the
> performance (switch to ext4, switch to mdbox), but I never had enough time
> to do that.
>
> At the moment I *need* more speed, we have too much waitI/O on the system
> and I already used all other performance and tuning-tricks (separated
cache,
> noatime, fsync and all that stuff).
>
> I have to change my setup, maybe somebody else here have hard facts:
>
> *) Is ext4 faster? How much faster?
> *) Is it faster because of the ext4 kernel-module (which can be used on
ext3
> to) or because of the ext4 filesystem layout?
>
>
> *) Is mdbox really faster? I'd like to have mdbox to have better
performance
> in running my backup-processes. But does it bring some performance boosts
> to?
>
>
> Thanks for any hints an tricks,
>
> Peer
>
>
* Morten Stevens <mstevens at imt-systems.com>:> We have switched our mailbox storage format from maildir to mdbox!I wonder how I can incrementally change over from Maildir to mdbox? I can of course use dsync to mirror Maildir: to mdbox:, but how can I make dovecot look at Maildir FIRST and (if that fails) at mdbox? (or vice versa). That would allow for a smooth transition... -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de
On 08.11.2011 15:16, Ralf Hildebrandt wrote:> * Morten Stevens <mstevens at imt-systems.com>: > >> We have switched our mailbox storage format from maildir to mdbox!I never tried it, but it should be possible to provide the mail_location from the user repsoitory (LDAP, SQL, whatever) So you can keep your global config, and use a script to convert one mailbox after another, and add a mail_location extra userdb field in the user repository to overwrite the global setting on a per-user-basis. Regards, Oliver
Hi there,>I never tried it, but it should be possible to provide the mail_location >from the user repsoitory (LDAP, SQL, whatever)Actually this works :-) Our userdb looks similar to: account1:xyz:000:000::/account1s/home/dir::userdb_mail=maildir:/account1s/home/dir/Maildir account2:xyz:000:000::/account2s/home/dir::userdb_mail=mdbox:/ account2s/home/dir http://wiki2.dovecot.org/UserDatabase/ExtraFields Concerning Maildir backups: what about a backup-to-disc-to-tape scheme using snapshots for the "to-disc" part and something like perpetual incrementals afterwards for the "top-tape" (secondary store) ? Regards Dirk -- Dirk Jahnke-Zumbusch Deutsches Elektronen-Synchrotron DESY IT Information Fabrics Member of the Helmholtz Association D-22603 Hamburg Notkestrasse 85 / 22607 Hamburg T: +49-40-899.81760 F: +49-40-899.41760 dirk.jahnke-zumbusch at desy.de>So you can keep your global config, and use a script to convert one >mailbox after another, and add a mail_location extra userdb field in the >user repository to overwrite the global setting on a per-user-basis. > >Regards, >Oliver
On 8.11.2011, at 15.50, Peer Heinlein wrote:> At the moment I *need* more speed, we have too much waitI/O on the system > and I already used all other performance and tuning-tricks (separated cache, > noatime, fsync and all that stuff).A few more ideas for Maildir if you haven't done yet: - maildir_very_dirty_syncs = yes - pop3_no_flag_updates = yes - Switching to dict file quota instead of Maildir++ quota.
On 11/8/2011 7:50 AM, Peer Heinlein wrote:> I have > 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA.That's a lot of mail (likely a large user base--not given), on a filesystem not designed for such, on a decent SAN controller--LUN RAID configuration not given.> I always wanted to make some mesurements about several influences to the > performance (switch to ext4, switch to mdbox), but I never had enough time > to do that.If you're going to switch filesystems, for this size dataset and concurrent workload, you're moving in the wrong direction.> At the moment I *need* more speed, we have too much waitI/O on the system > and I already used all other performance and tuning-tricks (separated cache, > noatime, fsync and all that stuff).EXT3/4 are not designed, nor optimized, for high concurrency workloads.> I have to change my setup, maybe somebody else here have hard facts: > > *) Is ext4 faster? How much faster?Simulated maildir workload test on 2.6.35-rc5, 128 threads (No data published for newer kernels): http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5_Mail_server_simulation._num_threads=128.html As you can see EXT4 shows a small gain over EXT3, ~20%. If you really want high performance it's time to move to XFS, properly configured to match the underlying RAID characteristics of the LUN(s) you're mounting. You'll prefer kernel 2.6.39+, 2.6.36 at minimum, so you get the delayed logging feature (2.6.35 had delayed logging but had problems in other areas). I'll assume with a >10TB mail store that you're seeing greater than 128 concurrent user operations regularly. As you can see from the graph, XFS will give you ~50% greater ops/s than EXT4 and ~90% greater than EXT3--yes, almost double that of EXT3. As the concurrency increases, so will this performance gap, as XFS was designed from day 1 for high concurrency workloads. This is a simulated mail server benchmark. However you should see similar gains with Dovecot. The XFS delayed logging feature will dramatically reduce the number of physical IOs required for journal writes (i.e. metadata IO), as will delayed allocation, a feature of XFS since its inception in 1994. EXT4 was the first of its lineage to gain delayed allocation, some 10+ years later, after Ted T'so studied the XFS code. In short, if you want an 'enterprise caliber' production Linux filesystem tailor made for high IO concurrency, XFS is it. JFS yields similar performance, but hasn't been actively developed for 8 years or so. XFS has substantial ongoing feature and fix development.> *) Is it faster because of the ext4 kernel-module (which can be used on ext3 > to) or because of the ext4 filesystem layout?AIUI, the bulk of the EXT4 performance advantage over EXT3 is the delayed allocation logic. The new EXT4 extent based on disk layout yields little in the way of additional performance, but much in free space management, fragmentation mitigation, etc.> *) Is mdbox really faster? I'd like to have mdbox to have better performance > in running my backup-processes. But does it bring some performance boosts > to?mdbox will substantially decrease physical IOs to your storage back end due to dramatically less metadata operations compared to maildir. You've stated you currently have a storage IOPS bottleneck, so I'd have to assume mdbox will seriously increase your overall performance. Good old mbox will do so as well, but everyone shuns it for various reasons, some valid, some not so valid. If you have an appropriate LUN available (sufficient size and spindle speed/count of member disks), properly create an XFS filesystem on it (read much before creating it), and moved to mdbox atop that, I think you'll be really surprised by how much you gain from simply changing filesystems and mailbox storage formats. If you double the size of the LUN you could potentially carry twice as many users with, fewer IOPS than you're seeing now, on essentially the same hardware platform. -- Stan