dovecot - Jun 2012 - best practises for mail systems

hello!

Can someone point me to some best practices in building high-available scalable
mail system or! share your own success stories.

 

I've read article in LJ "Building a Scalable High-Availability E-Mail
System with Active Directory and More"

but it seemed to be outdated and there's a single point of failure (Master
node).

 

What I want to achieve:

high-available,

horizontaly scalable,

with no single point of failure

mail solution.

 

Available hardware:

intel mfsys25 modular server with 2 storage controllers, 2 switches, 4 power
supply blocks

with

- 2 blade-servers in mfsys with:

2xIntel Xeon E5620 @ 2.40GHz with 8 cores each

- promise vtrak e610s (2 storage controllers, 2 power supply blocks)

- 6x 2TB SATA Hitachi HDS72302

 

We decided to go for KVM virtualization

and glusterfs for live migration for vm image but that's not what this is
all about :)

 

We installed centos on host systems.

 

for now while we could think of two ways to go:

 

The first way (currently at testing stage):

 

On each host system we created one VM and passed through 3x2TB disks into it.

 

In guests vms on top of this disks we made XFS and fired up glusterfs with
distributed replicated volumes for our mailstorage.

so it looks like this:

 

vm1??? replicate???? vm2

disk1 ------------> disk4

disk2 ------------> disk5

disk3 ------------> disk6

 

in each vm we mounted glusterfs and pointed dovecot to that dir for mail
creation (as ltmp) and imap4 user access.

also we use exim as smtp.

 

So, with glusterfs as mailstorage we can go for LVS to load balancing for exim
and dovecot.

so wherenever one of host systems (hence one of mail vms) goes down, users
don't notice that

'cause LVS points them to working smtp and imap4 servers 

and they get their mail 'cause of glusterfs.

 

Pros:

- high-available

- horizontaly scalable

- with no single point of failure

 

Cons:

- not quite sure if glusterfs is production ready solution 'cause I've
experienced split-brains during setting it up

- IO performance issue. Though we didn't yet run any io tests, but glusterfs
uses fuse to mount on clients. And guys on #gluster told me writing to the
glusterfs mount will not be strictly local io.

 

The second way:

 

split up the users mail with:

 

two back-end VMs each other on DIFFERENT host system with

- fat mailstorage with raid1+linear mode (mdadm)+XFS

- dovecot/exim-back-ends

 

and

 

two VMs for nginx-based proxy servers for imap4 and smtp - nginx can redirect
user to right back-end through HTTP-php-based logic.

 

Pros:

- we split up not only load for exim/dovecot but users mail IOs too

- no split-brains

 

Cons:

- If one of the host systems (hence one of back-end VMs with storage) goes down,
half of our users is unhappy

 

P.S. Sorry if this place is way wrong to ask for such things.

On 5.6.2012, at 6.14, ???????? ????????? ?????????? wrote:
> - not quite sure if glusterfs is production ready solution 'cause
I've experienced split-brains during setting it up
Last I've heard glusterfs causes corruption problems with Dovecot. You
should try stress testing it with imaptest: http://imapwiki.org/ImapTest

On 2012-06-05 05:14, ???????? ????????? ??????????
wrote:
> On each host system we created one VM and passed through 3x2TB disks into
it.
> 
>  
> 
> In guests vms on top of this disks we made XFS and fired up glusterfs with
distributed replicated volumes for our mailstorage.
> 
> so it looks like this:
> 
>  
> 
> vm1    replicate     vm2
> 
> disk1 ------------> disk4
> 
> disk2 ------------> disk5
> 
> disk3 ------------> disk6
> 
>  
> 
> in each vm we mounted glusterfs and pointed dovecot to that dir for mail
creation (as ltmp) and imap4 user access.
> 
> also we use exim as smtp.
> 
>  
> 
> So, with glusterfs as mailstorage we can go for LVS to load balancing for
exim and dovecot.
> 
> so wherenever one of host systems (hence one of mail vms) goes down, users
don't notice that
> 
> 'cause LVS points them to working smtp and imap4 servers 
> 
> and they get their mail 'cause of glusterfs.
> [...]
> Cons:
> 
> - not quite sure if glusterfs is production ready solution 'cause
I've experienced split-brains during setting it up
> 
> - IO performance issue. Though we didn't yet run any io tests, but
glusterfs uses fuse to mount on clients. And guys on #gluster told me writing to
the glusterfs mount will not be strictly local io.
I'm not familiar with LVS, but from the project description it seems
that you need a "front server" that does the load balancing, so you
either have to run at least two of these servers in parallel or add to
your cons that you introduced a single point of failure. But you
mentioned that you only have two servers, so you really can do this.

I would rather ensure high availability by running the two servers as
masters and using either IP address takeover or DNS failover (with
dynamic DNS) and either use Dovecot's replication (I haven't tested it
yet and I'm not sure what happens in case of IP address takeover) or a
file system that can handle these kinds of errors (e.g. Coda). You could
do load balancing via round-robin DNS. This only protects you against
the failure of single machine and because IMAP sessions are not
replicated between the servers, connections will get reset if one server
fails, but it's cost-effective and uses software that already exists.

Regards,
Matthias-Christian