dovecot - Feb 2010 - So, what about clustering and load balancing?

Hello world,

so, with beta2 of Dovecot 2.0 being available, what is the preferred
way to achieve load balancing and fault tolerance? As far as I can see
it, there are basically two options:

1) Use a HA shared storage, export either a cluster filesystem or NFS,
and have your dovecot servers mount that file system. Load balance
these servers (Cisco ACE, ldirectord, ...) and there you go. Overall
performance is limited by the speed of the shared storage and may
further decrease due to locking issues, imperformant cluster
filesystems and so on.

2) Set up every dovecot server with local storage and trigger a dsync
after each change (pyinotfiy, incron, parse LMTP delivery logs, ...).
Load balance these servers as before. Depending on the number of syncs
that have to be done, mailbox replication may lag behind. The need to
constantly spawn dsync processes may further decrease performance if
you don't do it right (stickiness of established connections,
replicating at most every xyz seconds).


What are your opinions on that matter?


Stefan

P.S: I've set up option two in a test setup, though the incron/inotify
part is still giving me a headache.

On 13.2.2010, at 17.48, Stefan Foerster wrote:
> 1) Use a HA shared storage, export either a cluster filesystem or NFS,
> and have your dovecot servers mount that file system. Load balance
> these servers (Cisco ACE, ldirectord, ...) and there you go.
With NFS if you want to avoid random errors, you still need to do the load
balancing in a way that user's mails are never accessed simultaneously from
two servers at the same time. See recent "quick question" thread about
this. Cluster filesystems should be able to handle this better, although for
performance it's probably still a good idea to do this.

> 2) Set up every dovecot server with local storage and trigger a dsync
> after each change (pyinotfiy, incron, parse LMTP delivery logs, ...).
> Load balance these servers as before. Depending on the number of syncs
> that have to be done, mailbox replication may lag behind. The need to
> constantly spawn dsync processes may further decrease performance if
> you don't do it right (stickiness of established connections,
> replicating at most every xyz seconds).
The above solution would require some new development to create such a daemon
that keeps track of changes and dsyncs users when necessary. The fastest way
would probably be to create a plugin that uses notify plugin's events and
sends them to the daemon.

dsync process creation could be avoided by making dsync another service that
listens on unix socket. Wouldn't be a big change. v2.0 makes this really
easy :)

Also 3) Use DRBD server pairs.

I would suggested AFS as shared storage, with user-specific DNS
based load balancing. (user connects to 'username.imap.example.com')
This probably falls over with huge shared mailboxes though.

If you give the IMAP servers sufficient AFS cache space, any data
that needs to be read is sitting hot in the AFS cache, and all writes
are duplicated back to the AFS server. The ability of afs to take
snapshots of live volumes is quite nice for backups..
http://docs.openafs.org/AdminGuide/ch05s06.html


On Sat, Feb 13, 2010 at 04:48:07PM +0100, Stefan Foerster
wrote:
> Hello world,
> 
> so, with beta2 of Dovecot 2.0 being available, what is the preferred
> way to achieve load balancing and fault tolerance? As far as I can see
> it, there are basically two options:
> 
> 1) Use a HA shared storage, export either a cluster filesystem or NFS,
> and have your dovecot servers mount that file system. Load balance
> these servers (Cisco ACE, ldirectord, ...) and there you go. Overall
> performance is limited by the speed of the shared storage and may
> further decrease due to locking issues, imperformant cluster
> filesystems and so on.
> 
> 2) Set up every dovecot server with local storage and trigger a dsync
> after each change (pyinotfiy, incron, parse LMTP delivery logs, ...).
> Load balance these servers as before. Depending on the number of syncs
> that have to be done, mailbox replication may lag behind. The need to
> constantly spawn dsync processes may further decrease performance if
> you don't do it right (stickiness of established connections,
> replicating at most every xyz seconds).
> 
> 
> What are your opinions on that matter?
> 
> 
> Stefan
> 
> P.S: I've set up option two in a test setup, though the incron/inotify
> part is still giving me a headache.
-- 
--------------------------------------------------------------------------
Troy Benjegerdes                'da hozer'                hozer at
hozed.org

Unless hours were cups of sack, and minutes capons, and clocks the tongues
of bawds, and dials the signs of leaping houses, and the blessed sun himself
a fair, hot wench in flame-colored taffeta, I see no reason why thou shouldst
be so superfluous to demand the time of the day.  I wasted time and now doth
time waste me.                        -- William Shakespeare