dovecot - Aug 2007 - Dovecot strong or not for a big Webmail architecture

Hello,

Do you think Dovecot could handle millions of active users in a big architecture
?

This cluster could be for example (each server is a bi quad Xeon 2.66 Ghz) :
- 40 Dovecot servers 
- 4 LVS
- 20 Apache+PHP
- 2 Openldap
- 20 Postfix + ClamAV + SpamAssassin
- 1 NFS Netapps

For millions of user it could be multiple different clusters of 40 Dovecot
servers with 1 Netapp for each cluster.
I want to work in NFS.
I've experimented problem with Cyrus and NFS on Netapp.
I want to use maildir, I can manage limitations of inodes with Netapp.

I know webmail.us use Dovecot, what is the most big dovecot architecture known ?
Do you think Dovecot can handle 1 million of active users in a good architecture
?
Do you think it's a good solution to use one synchronised local Openldap on
each server Dovecot ?
Do you think It's possible to use Postgresql or MySQL instead of Openldap ?
Are cyrus or courrier-imap better solutions ?
Someone say "Zimbra is highly scalable and fast", I think Zimbra could
be to heavy in this architecture, is Dovecot scalable ?

Thank you for your ideas.

Regards.

John Fistack


     
_____________________________________________________________________________
Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail

On Thu, 2007-08-02 at 19:06 +0000, John fistack wrote:
> For millions of user it could be multiple different clusters of 40 Dovecot
servers with 1 Netapp for each cluster.
> I want to work in NFS.
You'll want to read at least http://wiki.dovecot.org/NFS. Dovecot v1.1
makes problems with attribute/data caches go away.
> I know webmail.us use Dovecot, what is the most big dovecot architecture
known ?
Portugal Telecom is the largest installation I know of. They use NFS +
Netapps also.
> Are cyrus or courrier-imap better solutions ?
Cyrus doesn't work properly with NFS and I don't think Courier has any
advantages over Dovecot.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070802/ed984096/attachment-0002.bin>

Hello,

On Thu, 2 Aug 2007 19:06:05 +0000 (GMT) John fistack
<john.fistack at yahoo.fr> wrote:
> 
> Do you think Dovecot could handle millions of active users in a big
> architecture ?
> 
Sure, but the architecture will play a bigger role than just
Dovecot.
> This cluster could be for example (each server is a bi quad Xeon 2.66
> Ghz) :
> - 40 Dovecot servers 
> - 4 LVS
> - 20 Apache+PHP
> - 2 Openldap
> - 20 Postfix + ClamAV + SpamAssassin
> - 1 NFS Netapps
> 
I'm not sure where you are pulling these numbers from (but then again
webmail is a lot more resource intensive than real mail clients).
We are handling about 100k users with a total of 9 machines and one
SAN backend. And yes, this includes all the components up there with
Exim instead of Postfix. About 10-20% are using webmail. 6 machines
are cluster pairs, 3 are MXs (no need for clustering there) and yes,
any single machine can fail and things will still work, pretty much
w/o noticeable performance impact, too. The whole thing is designed
to be pretty much indefinitely scalable.

For historical reasons we stayed clear of NFS (this system architecture
has been in production for 7 years, the last 3 with Dovecot), so I
can not comment on that part from own experience. But I have a hard
time imagining any single storage backend not melting from the traffic
that these amounts of machines would generate (if they are actually 
needed). And even if the backend could handle it, the network towards
it would be saturated a long time before those 40 Dovecot servers would
run out of steam CPU wise.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                NOC
chibi at gol.com   	Global OnLine Japan/Fusion Network Services
http://www.gol.com/

On Thu, August 2, 2007 3:06 pm, John fistack <john.fistack at yahoo.fr>
said:
> I know webmail.us use Dovecot, what is the most big dovecot architecture
known ?
> Do you think Dovecot can handle 1 million of active users in a good
architecture ?
Yep... and we have 500K very active users on it.  We've scaled Dovecot
horizontally without NFS, just lots of independent Dovecot server intances with
smart proxy/mail-routing infrastructure around it.  It can easily scale to
millions of users this way.  Just make sure you throw enough disks at it and
you'll be fine, with or without NFS.
> Do you think it's a good solution to use one synchronised local
Openldap on each
> server Dovecot ?
No.  I'd replicate your ldap database on a few servers that are dedicated to
that purpose, and on each mail server use Dovecot's auth_cach feature to
minimize how often it needs to query ldap.
 
> Do you think It's possible to use Postgresql or MySQL instead of
Openldap ?
We use MySQL.  MySQL handles frequent writes better than OpenLDAP from our
experience.  It is also simpler for us to do replication and troubleshooting
because we employ several MysQL gurus already.
> Are cyrus or courrier-imap better solutions ?
Definitely not courier-imap, because of it's lack of indexes.  Not sure
about Cyrus.
> Someone say "Zimbra is highly scalable and fast", I think Zimbra
could be to heavy
> in this architecture, is Dovecot scalable ?
My opinion is Zimbra is too heavy beccause of the way mail is stored on the
backend.  Dovecot with maildir scales out well, and the promise of dbox mail
storage format appears that it will make it even more scalable down the road.

Bill