dovecot - Jul 2010 - Proxy IMAP/POP/ManageSieve/SMTP in a large cluster enviroment

Hi to all in the list, we are trying to do some tests lab for a large scale
mail system having this requirements:
- Scale to maybe 1million users(Only for testing).
- Server side filters.
- User quotas.
- High concurrency.
- High performance and High Availability.

We plan to test this using RHEL5 and maybe RHEL6.

As a storage we are going to use an HP EVA 8400 FC(8 GB/s)

We defined this functional roles.

1- Outgoing SMTP gateway servers.

 - Load balanced with Piranha LVS using Direct Routing.
 - ( n servers)RHEL5/6 using Postfix Latest version Amavisd-new Clamav
Spamassasin maybe some other filters.

2- Incomming SMTP gateway servers.

 - Load balanced with Piranha LVS using Direct Routing.
 - RHEL5/6 using Postfix Latest version Amavisd-new Clamav Spamassasin( n
servers) maybe some other filters.

3- Webmail Cluster farm.

 - Load balanced with Piranha LVS using Direct Routing.
 - ( n servers)RHEL5/6 using RHCS Active/Active. Apache2 PHP5 Horde.

4- Openldap load balanced openldap cluster.
 - Load balanced with Piranha LVS using Direct Routing.
 - ( n servers)RHEL5/6 using Openldap.

5- IMAP/POP/ManageSieve/SMTP Proxy or Director.
 - Load balanced with Piranha LVS using Direct Routing.
 - ( n servers)RHEL5/6 using Dovecot - Postfix(Director or Proxy).

6- Mail backend.
 - ( n servers)RHEL5/6 using Dovecot.

Now for functional role 6 "Mail Backend" we have some dilemmas.
   - Recommended scalable filesystem to use for such scenario(Clustered or
not).
     -GFS2?? We have very bad experiences with GFS1 and maildir, GFS2
doesn't seems to improve this also. Using  some techniques for session
afinity with a backend server seems to help with the locking problems of GFS
and the cache. Using GFS many IMAP SMTP servers can in parallel write or
read to user mailboxes, if GFS can perform well we prefer this, will have to
proxy o use director for the cache problem.
     - Ext4, Ext3 well tested with many setups but only one backend server
can access one ext4 lun, so we have to proxy a user always to a ip backend
address or VIP in a case of failure cluster software has to move the server
to another node, the cluster service is formed by a IP Address and one or
many Luns wich moves from node to node in a case of failure.

  - Recommended scalable method  for partition divition.
     - Directory Hashing(How it actually works the theory behind this), we
see the wiki but we need to understand the theory to balance the directory
structure.
     - Using some criteria ex Lun1(a-h users), Lun2(i-m users)....

 - Proxy or Director is a must for a Clustered Filsystem or for the solution
of a user belonging to one server.
     - In case we use a Proxy with dovecot we know we can use it with
IMAP/POP but are not sure for ManageSieve and more important for SMTP. The
question is, how can we proxy for the same user IMAP/POP/ManageSieve and
SMTP delivery to mailbox at the same time, can the incoming mail gateways
send the email to the proxy servers and dovecot proxy the request for smtp
or lmtp appply the same proxy criteria that uses for the protocols IMAP and
POP and send the email to the correct backend server and this server do the
final delivery?? I mean proxy all the protocols for a user
IMAP/POP/SMTP-LMTP?? Any example...

Right now we have doubts on this stuff, we are really going to appreciate
any help or advice on this...
best regards

Hello,

just my comment on you topic: if I'd design such setup, I'd expect, that
sooner or later some additional flexibility is needed, that there will
be same tasks you never thought about in advance. I'm not sure, if in
such case I'd rely on Postfix. Postfix might be fast, but it is by no
means as flexibile as Exim is. Missing performance (of Exim, if this
matters) you can buy using more servers. Missing flexibility you can't
get easily.

Just my two cents - and because you asked ?
-- 
Heiko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100719/32384494/attachment-0002.bin>

Aliet Santiesteban Sifontes put forth on 7/18/2010 4:22 PM:
> 6- Mail backend.
>  - ( n servers)RHEL5/6 using Dovecot.
> 
> Now for functional role 6 "Mail Backend" we have some dilemmas.
>    - Recommended scalable filesystem to use for such scenario(Clustered or
> not).
You have 3 choices:  NFS, a cluster filesystem, or each of those with VMWare
ESX.  I recommend the cluster filesystem solution, specifically SGI CXFS, atop
ESX (if you can afford it):

http://www.sgi.com/pdfs/2816.pdf
http://www.vmware.com/products/vsphere/esxi-and-esx/index.html
>      -GFS2?? We have very bad experiences with GFS1 and maildir, GFS2
> doesn't seems to improve this also. Using  some techniques for session
> afinity with a backend server seems to help with the locking problems of
GFS
> and the cache. Using GFS many IMAP SMTP servers can in parallel write or
> read to user mailboxes, if GFS can perform well we prefer this, will have
to
> proxy o use director for the cache problem.
CXFS doesn't suffer these problems associated with GFS1/2.
>      - Ext4, Ext3 well tested with many setups but only one backend server
> can access one ext4 lun, so we have to proxy a user always to a ip backend
> address or VIP in a case of failure cluster software has to move the server
> to another node, the cluster service is formed by a IP Address and one or
> many Luns wich moves from node to node in a case of failure.
Again, CXFS eliminates this problem as all clients "own" all LUNs all
the
time.  Or are you talking about booting the cluster member machines from SAN
LUNs, i.e. "Boot from SAN", /boot and / reside on a SAN LUN?  In that
case,
using VMWare ESX server with High Availability and Vmotion to move or restart
failed cluster guests would be a much better more manageable solution as each
guests' system disk would reside within a VMWare virtual disk which would
physically be a single SAN LUN.  You could stick 100 or 1000 virtual machine
disk images on that single LUN, each of them being the boot disk for one of
your mail cluster hosts.
>   - Recommended scalable method  for partition divition.
>      - Directory Hashing(How it actually works the theory behind this), we
> see the wiki but we need to understand the theory to balance the directory
> structure.
>      - Using some criteria ex Lun1(a-h users), Lun2(i-m users)....
You're talking about the mail store directory structure for the IMAP servers
here right?  I wouldn't do the split you describe above.  It will cause more
problems than it will solve.  Again, you can partition and format a single SAN
LUN, say 10TB, with CXFS and have all back end IMAP server hosts mount this
same filesystem.  You can put well over 1 million directories and files within
this CXFS filesystem without seeing performance degradation due to CXFS
overhead.
>  - Proxy or Director is a must for a Clustered Filsystem or for the
solution
> of a user belonging to one server.
>      - In case we use a Proxy with dovecot we know we can use it with
> IMAP/POP but are not sure for ManageSieve and more important for SMTP. The
You should try to proxy all IMAP/POP client connections to the same Dovecot
server, not necessarily from a data integrity/file locking standpoint (if
using CXFS locking issues are moot), but more from a performance and
efficiency standpoint due to Dovecot indexing.

Regarding inbound SMTP, you don't need to worry about locking here either.
You will have both, or 4, or however many MX hosts connected to the SAN and
all of them will run the CXFS client, Postfix and Dovecot/LDA/ManageSieve.
They will all mount the same SAN LUN that the backend IMAP servers do, and
Postfix/Dovecot LDA will drop the mail into the appropriate user directory
after Sieve scripts are run.  You will create a directory within the user mail
storage LUN where you will store ManageSieve scripts.  This allows for all
inbound SMTP servers to read/write the directory where the ManageSieve scripts
are kept.  Thus, you'll run ManageSieve on all your MX hosts.  It won't
matter
if you proxy MS operations or not because they are so infrequent and you'll
not have any locking issues.  But by running MS on all the MX hosts you enable
redundancy.  The only thing you need a director for in this case is routing
requests around a downed MX host.
> question is, how can we proxy for the same user IMAP/POP/ManageSieve and
> SMTP delivery to mailbox at the same time, can the incoming mail gateways
> send the email to the proxy servers and dovecot proxy the request for smtp
> or lmtp appply the same proxy criteria that uses for the protocols IMAP and
> POP and send the email to the correct backend server and this server do the
> final delivery?? I mean proxy all the protocols for a user
> IMAP/POP/SMTP-LMTP?? Any example...
No.  See above, and, IMAP/POP requests would be proxied to the backend IMAP
servers.  Inbound SMTP and ManageSieve requests should be routed to the MX
hosts, no need to proxy or balance them with any affinity.  This is because
SMTP/ManageSieve packets aren't stateful in nature like IMAP/POP are.
> Right now we have doubts on this stuff, we are really going to appreciate
> any help or advice on this...
> best regards
I'm not an expert on all this stuff, and I may not be 100% accurate with all
my advice.  I'm much more knowledgeable on the fiber channel SAN, filesystem
(XFS/CXFS), and VMware stuff than Dovecot and IMAP clustering specifically.  I
hope I've at least helped point you in a good direction in some small way.

-- 
Stan

On Sun, 2010-07-18 at 18:22 -0300, Aliet Santiesteban Sifontes
wrote:
> 
>      - In case we use a Proxy with dovecot we know we can use it with
> IMAP/POP but are not sure for ManageSieve and more important for SMTP. The
> question is, how can we proxy for the same user IMAP/POP/ManageSieve and
> SMTP delivery to mailbox at the same time, can the incoming mail gateways
> send the email to the proxy servers and dovecot proxy the request for smtp
> or lmtp appply the same proxy criteria that uses for the protocols IMAP and
> POP and send the email to the correct backend server and this server do the
> final delivery?? I mean proxy all the protocols for a user
> IMAP/POP/SMTP-LMTP?? Any example...
ManageSieve can be proxied exactly the same as IMAP/POP3. Use LMTP for
mail delivery and then you don't need to worry about SMTP. So you can
use Dovecot proxying for IMAP, POP3, ManageSieve and LMTP.