dovecot - Sep 2011 - is it necessary lmtp and director to avoid index corruption in mail delivery?

Hello all,
If I have several postfix/dovecot-lda boxes with shared nfs storage, how
director helps in this scenario? is it necesary to use lmtp instead of
dovecot-lda?


With postfix/dovecot-lda boxes incoming mail happens at the smtp layer but
director redirects are working in the lmtp/imap/pop3 layer.

On Wed, Sep 07, 2011 at 11:26:28AM +0100, mailing lists
wrote:
> Hello all,
> If I have several postfix/dovecot-lda boxes with shared nfs storage,
> how director helps in this scenario?
The director can help by directing each unique user to the same backend
server for each delivery, which should give you better performance
(indexes cached in memory on only one machine instead of on all,
inotify will work for IMAP NOTIFY), and also avoid some apparent race
conditions that has lead to index corruption when several nodes are
writing to the same mailbox at the same time.
> is it necesary to use lmtp instead of dovecot-lda?
Yes.

> With postfix/dovecot-lda boxes incoming mail happens at the smtp layer
> but director redirects are working in the lmtp/imap/pop3 layer.
Right.

With LMTP you can have dovecot listening on the network (port 24/tcp)
for incoming mail. No need for postfix on the backend mailstorage
servers.

Our setups has been:

	Mailgw[1-14] ---smtp--> dovecot-server[1-5] (postfix + dovecot/lda)

and used MX priorities to make all deliveries go to the same 
dovecot-server with the others as backup.

	$ dig mx deliver.example.com +short
	10 dove2.example.com.
	15 dove3.example.com.
	20 dove4.example.com.
	25 dove5.example.com.
	5 dove1.example.com.


Now I'm moving towards:

	Mailgw[1-14] ---lmtp--> dovecot-server[1-5] (dovecot/lmtp)

but worry that the index corruption might hit me again.. Ideally
I want:

	Mailgw[1-14] ---lmtp--> directors --lmtp--> dovecot-server[1-5]
(dovecot/lmtp)

but lmtp-proxying seems to have bugs (ref: my latest mails to this
list).


  -jf

JF, thank you for the reply.

I'm just curious how to big players fix this problem, which seems impact
systems with shared storage.

Have a nice day.



On 09/07/2011 12:59 PM, Jan-Frode Myklebust wrote:
> On Wed, Sep 07, 2011 at 11:26:28AM +0100, mailing lists wrote:
>> Hello all,
>> If I have several postfix/dovecot-lda boxes with shared nfs storage,
>> how director helps in this scenario?
> 
> The director can help by directing each unique user to the same backend
> server for each delivery, which should give you better performance
> (indexes cached in memory on only one machine instead of on all,
> inotify will work for IMAP NOTIFY), and also avoid some apparent race
> conditions that has lead to index corruption when several nodes are
> writing to the same mailbox at the same time.
> 
>> is it necesary to use lmtp instead of dovecot-lda?
> 
> Yes.
> 
> 
>> With postfix/dovecot-lda boxes incoming mail happens at the smtp layer
>> but director redirects are working in the lmtp/imap/pop3 layer.
> 
> Right.
> 
> With LMTP you can have dovecot listening on the network (port 24/tcp)
> for incoming mail. No need for postfix on the backend mailstorage
> servers.
> 
> Our setups has been:
> 
> ??? Mailgw[1-14] ---smtp-->? dovecot-server[1-5] (postfix + dovecot/lda)
> 
> and used MX priorities to make all deliveries go to the same
> dovecot-server with the others as backup.
> 
> ??? $ dig mx deliver.example.com +short
> ??? 10 dove2.example.com.
> ??? 15 dove3.example.com.
> ??? 20 dove4.example.com.
> ??? 25 dove5.example.com.
> ??? 5 dove1.example.com.
> 
> 
> Now I'm moving towards:
> 
> ??? Mailgw[1-14] ---lmtp-->? dovecot-server[1-5] (dovecot/lmtp)
> 
> but worry that the index corruption might hit me again.. Ideally
> I want:
> 
> ??? Mailgw[1-14] ---lmtp-->? directors --lmtp-->? dovecot-server[1-5]
(dovecot/lmtp)
> 
> but lmtp-proxying seems to have bugs (ref: my latest mails to this
> list).
> 
> 
>??? -jf