dovecot - Jul 2011 - Performance with 200k messages in Maildir

What should I expect performance-wise if I put 200000 messages in a
Maildir store and use two or three MUAs (mutt and Thunderbird), on an
Athlon dual core 2GHz with SATA drives in software RAID (Linux)?

Like; would it be useless/crawling, usable or pretty fast. I imagine MUA
startup / first time folder read would be slow, but daily use more or
less ok.

Thanks to Dovecot's indexing, I imagine I would be better off
interfacing Mutt to Maildir via IMAP and Dovecot rather than accessing
Maildir/ directly?

-- 
Vegard Svanberg <vegard at svanberg.no> [*Takapa at IRC (EFnet)]

To get benefits of Dovecot indexing you would need to interface Mutt to IMAP or
else you will scan the Maildir directory
each time Mutt is started.

You need to know more info than just 200,000 messages.

How many mailboxes, users, messages per day etc

Vegard Svanberg wrote, On 18/07/2011 11:27:
> What should I expect performance-wise if I put 200000 messages in a
> Maildir store and use two or three MUAs (mutt and Thunderbird), on an
> Athlon dual core 2GHz with SATA drives in software RAID (Linux)?
>
> Like; would it be useless/crawling, usable or pretty fast. I imagine MUA
> startup / first time folder read would be slow, but daily use more or
> less ok.
>
> Thanks to Dovecot's indexing, I imagine I would be better off
> interfacing Mutt to Maildir via IMAP and Dovecot rather than accessing
> Maildir/ directly?
>

I agree with yr points on TBird, moving large amounts of messages can cause it
to hang with CPU pegged at max for ages.
TBird v2 was nice and nippy, v3 acceptable, v4/v5 are just awfully slow overall.
TBird uses mbox storage format which probably stuffs it up on large
deletes/moves etc.

Just did a count on our server, 350G of email (largest single mailbox is 40G,
that is 350k messages), total messages is
3.6mil+, biggest problem is on backup, ive read that the latest rsync has fast
start now rather than wait to finish
scanning.
Im intrested in the latest mdbox format to reduce how many files we have.
Try backing up small files fast enough to LTO5, tar it all up first before
backup I think.
Ile move all our maildirs to 10k SAS soon hopefully to lower the load on the
SATA disks.

Thomas Harold wrote, On 19/07/2011 16:03:
> On 7/19/2011 5:54 AM, Ricardo Branco wrote:
>> If you have 200k all within one folder progs like TB will have issues
>> loading it all up and may hang when you try to do moves/deletes etc,
not
>> sure if mutt stores a local cache of headers, thats the biggest worry.
>> Biggest single folder ive seen at our office had 60k messages, it loads
>> slowly on a cold cache in TB.
>> Biggest mailbox has over 350k, my mailbox is over 250k, ofcourse thats
>> across several folders.
>> Mailserver is on VMware server (local drives), datastore (with
maildirs)
>> is separate NFS server on 11x2TB SATA R6 array (has other SAS disks for
>> other things).
>> If you have it spread out in different folders then it wont be so bad.
>>
>
> 60k in a single folder is about the upper limit for TBird (TBird v2 was
actually better suited for this).  But
> drag-n-drop breaks if you try to do more then 3-5k messages at a time. 
When a mailbox gets over 30-50k messages, I
> archive some of them off to a sub-folder in Thunderbird. One of my TBird
mailboxes is about 880,000 messages, almost
> 6GB of email, spread across dozens of directories.
>
> Assuming MailDir storage, the bigger issue will be (a) how well the
filesystem handles tens of thousands of files in a
> single folder (b) the physical disks / speed / number of spindles (c) how
busy the CPU is on the server and maybe (d)
> the amount of server RAM that can be used as cache/buffer.  Ext3 is
probably fine as long as directory indexing is
> turned on, but ext4 might be better (or something else that deals well with
lots of small files).
>
> The other side is how fast the disks are on the local client.  An SSD drive
or 10k RPM drive on the local desktop
> helps a lot when you get up into the larger mailboxes.

On 18/07/2011 11:27, Vegard Svanberg wrote:
> What should I expect performance-wise if I put 200000 messages in a 
> Maildir store and use two or three MUAs (mutt and Thunderbird), on an 
> Athlon dual core 2GHz with SATA drives in software RAID (Linux)?
>
> Like; would it be useless/crawling, usable or pretty fast. I imagine 
> MUA startup / first time folder read would be slow, but daily use more 
> or less ok.
>
> Thanks to Dovecot's indexing, I imagine I would be better off 
> interfacing Mutt to Maildir via IMAP and Dovecot rather than accessing 
> Maildir/ directly? 
200,000 messages under Maildir in a single folder.

You will tend to have a relatively large number of inodes compared to 
the relative overall scale of the system.

This will tend to make backing-up a nuisance.

One thing you could consider is storing your mail under "mdbox". This 
should drastically reduce the number of inodes.

Just remember that under "mdbox" the so-called "indexes" are
actually
critical data files (i.e. they cannot be re-created like they could be 
with Maildir or mbox).

Bill