dovecot - Apr 2011 - Questions about converting maildir to mdbox.

First of all i would like to thanks Timo for the excellent job done in
dovecot.

We are using it in you prodution server with about 5 millions of emails and
it is pretty good, only a performance problens but i guess it is realated to
ocfs2 that we are using.

So lets begin:

We are using dovecot 2.0.6 with maildir in an ocfs2 partiton.

It is pretty slow access in peak time, but it is related to ocfs2 as i said
before, i am trying to mimimize the ocfs2 latancy by using mdbox. Another
problem that we have are Full backup that takes about 24h and incremental
takes about 13h

I am thinking in converting some email for testing propose, but i am
wondoring how would dovecot convert this, like if i convert all mail from
maildir to mdbox it would make many mailbox based on this seetings ? Or in
the first convert it would make a big mdbox file and after it it will rotate
by day or it would make many boxes just based on size?

mdbox_rotate_size = 5m
mdbox_rotate_interval = 1d

Well, another thing can i make rotate interval happens in a certain time of
day? end from where it takes this time ? it rotates at midnight ?

Another one!, if i lose the index files what really happens ? Sometimes in
my server the index get corrupet and as i am reading mdbox index files held
information that is not in other place. So what should i do to prevent any
possible lose of data even if i lose the indexes?

I was reading about backup restore also, but i did not get it pretty well.
With maildir we simple restore the files and the index gets rebuilt! What
should i do with mdbox ? Restore the files and the index ? But how about the
other emails that the account got after the backup, how i should merge the
multiple files and more importanet how do i merge the index files ?

Well, sorry about my english!

And Thanks alot guys!

[]'sf.rique

Henrique Fernandes put forth on 4/10/2011 12:29 PM:
> We are using dovecot 2.0.6 with maildir in an ocfs2 partiton.
> 
> It is pretty slow access in peak time, but it is related to ocfs2 as i said
> before, i am trying to mimimize the ocfs2 latancy by using mdbox. Another
> problem that we have are Full backup that takes about 24h and incremental
> takes about 13h
Last you posted here you didn't have enough spindles in the array for
the size of the workload, and you didn't have a dedicated quality GbE
switch for strictly OCFS traffic between Dovecot hosts.  Has any of this
been rectified?  If not, changing mailbox formats isn't going to fix
your problems.  If you really want to test this theory to the limit,
converting to mbox (yes, good old mbox, without the d) will reduce
metadata load more than mdbox, by a huge margin.

Given the constraints on your ability to reconfigure the disks in the
EMC array (as someone else owns it and you're borrowing space) to meet
your needs, and your apparent lack of funding allowing purchasing your
own array, the things you should really do to fix most of your problems are:

1.  Stick with maildir
2.  Configure a Dovecot director box to achieve stickiness
3.  Put a quality 200GB SSD drive in each Dovecot IMAP server
    (smaller if you can get away with it to save $$, larger if
     if 200GB isn't enough)
4.  Change the index location on each server to the local SSD drive

Over time the user indexes will be gradually rebuilt onto the local fast
SSD drive.  This will eliminate the bulk of the IOPS load on OCFS, and
should fix your performance issues pretty thoroughly.  Well, except for
the slow backup.  The only way around that is more spindles in the EMC
array LUN where mailboxes reside.


-- 
Stan

On Sun, 2011-04-10 at 14:29 -0300, Henrique Fernandes wrote:
> I am thinking in converting some email for testing propose, but i am
> wondoring how would dovecot convert this, like if i convert all mail from
> maildir to mdbox it would make many mailbox based on this seetings ? Or in
> the first convert it would make a big mdbox file and after it it will
rotate
> by day or it would make many boxes just based on size?
> 
> mdbox_rotate_size = 5m
This setting affects all saves, so dsync doesn't create files larger
than 5 MB (although if you have over 5 MB mail then it goes fully into
the file).
> mdbox_rotate_interval = 1d
All mails are saved within same day so this makes no difference.
> Well, another thing can i make rotate interval happens in a certain time of
> day? end from where it takes this time ? it rotates at midnight ?
It's at midnight currently, yes. Maybe there should be a setting for
this.
> Another one!, if i lose the index files what really happens ? Sometimes in
> my server the index get corrupet and as i am reading mdbox index files held
> information that is not in other place. So what should i do to prevent any
> possible lose of data even if i lose the indexes?
Don't lose indexes :) mdbox anyway tries really hard to fix corrupted
indexes. It also keeps a backup index file around which it can use. So a
"corrupted index" typically doesn't lose (almost) any data, but if
the
entire index file (and its backup) gets lost, then there's nothing
Dovecot can do. Keep backups.
> I was reading about backup restore also, but i did not get it pretty well.
> With maildir we simple restore the files and the index gets rebuilt! What
> should i do with mdbox ? Restore the files and the index ? But how about
the
> other emails that the account got after the backup, how i should merge the
> multiple files and more importanet how do i merge the index files ?
I guess you're talking about "doveadm import" command. It copies
messages from a given message store to destination. You shouldn't ever
touch filesystem directly with mdbox, everything should be done via
doveadm command.

If you've lost one user's all mails, you can just "doveadm
import"
everything back there. The mails may be out of order with some clients,
but most support sorting by (received) date so it shouldn't matter that
much..

I guess you could also 1) restore the mdbox to /tmp from backup, 2) stop
mail deliveries to the user, 3) doveadm import the new mails to /tmp, 4)
rm -rf new mails and mv all the mails from /tmp.

Something like that..