dovecot - Jun 2011 - dovecot, mdbox and indexes

Hello everybody,

	We are updating our old dovecot 1.1 servers to 2.0. We have a farm os 
servers, with users in nfs filesystems and indexes in local fs. Mail is 
stored in maildir format. We don't warrant that a user is always 
directed to the same server, although our load balancer sends all 
request from one IP to the same server (during a session time with a 
inactivity timeout). This is the second scenario described at 
http://wiki2.dovecot.org/NFS.

	We are now trying to migrate to the third scenario and using mdbox.

	As an intermediate step, we have configured this second scenario, but 
with indexes in NFS filesystems. The configuration is:

* mmap_disable = yes
* dotlock_use_excl = no
* mail_nfs_storage = yes
* mail_nfs_index = yes
* lock_method = fcntl (we have also tried with dotlock)

	This configuration works without any problems with few users, but with 
high load, we have a lot of lock timeouts error like:

Jun 29 10:24:04 myotis34 dovecot: imap(mpm): Error: Timeout (180s) while 
waiting for lock for transaction log file 
/home/pas/01/014601/maildir_indexes/.INBOX/dovecot.index.log

	These errors cause a corrupted index and we have to manually delete the 
user's index so he could read his mail again.

	So we are planning to configure the third scenario. In this 
configuration, we don't have any problem during our tests if we use 
maildir format. But if we use mdbox one, we have problems when we have 
to change the final server for the server (We allways send the user to 
the same final server, but sometimes, if the server is down, for 
example, this is not possible). The tests we have done (with POP) are:

* I connect to "my" server. There's no problem. I have my mail. I
delete
a mail.
* I connect then to a server that have no index for me. The only problem 
is that the previously deleted mail appears again.
* I connect then to a server with an outdated index: this is the 
problem. I have only the messages in that outdated index. The mail 
received since then has disappeared (although I could receive new mail).

	How could I solve this problem? Is this really a problem? Should I use 
a shared storage for indexes? What is the best configuration for a high 
availability service?

-- 
Angel L. Mateo Mart?nez
Secci?n de Telem?tica
?rea de Tecnolog?as de la Informaci?n       _o)
y las Comunicaciones Aplicadas (ATICA)      / \\
http://www.um.es/atica                    _(___V
Tfo: 868887590
Fax: 868888337

On 30/06/2011 13:28, Angel L. Mateo wrote:
>     We are now trying to migrate to the third scenario and using mdbox.
>
>     These errors cause a corrupted index and we have to manually 
> delete the user's index so he could read his mail again.
>
> * I connect then to a server that have no index for me. The only 
> problem is that the previously deleted mail appears again.
> * I connect then to a server with an outdated index: this is the 
> problem. I have only the messages in that outdated index. The mail 
> received since then has disappeared (although I could receive new mail).
>
>     How could I solve this problem? Is this really a problem? Should I 
> use a shared storage for indexes? What is the best configuration for a 
> high availability service?
>
!!!!! DANGER, DANGER !!!!!!

Under mdbox, the so-called "indexes" are not indexes in the
traditional
(database) sense of the word (a file maintained alongside the main data 
file which is used to speed up access to the main file, and can be 
regenerated at will), but rather are (binary) database files which 
contain crucial metadata which is not held anywhere else.

These filed cannot be regenerated at will.

For example:

The so-called "map index" (message store database) contains a (flat) 
list of all messages in a (user's) mailstore together with their 
reference counts, which storage file they're stored in.

The "folder index" (folder message metadata database) in each folder 
directory contains a list of the message numbers which are "in" that 
folder. Those message numbers identify a particular message in the 
message store database, which identify a particular m.* storage file, 
and where within that file the message is.

Lose these files, and you destroy the information about:
* which message is supposed to be in which folder,
* which messages are supposed to be deleted and which aren't, and
* all message keywords.

Under mdbox, losing these files buggerises the mailstore.

You may wish to read http://wiki2.dovecot.org/MailboxFormat/dbox

"... with dbox the Index files actually contain significant data which 
is held nowhere else. Index files for both *single-dbox* and 
*multi-dbox* contain message flags and keywords. For *multi-dbox*, the 
index file also contains the map_uids which link (via the "map index")
to the actual message data. This data cannot be automatically recreated, 
so it is important that Index files are treated with the same care as 
message data files."

Regards,

Bill