dovecot - Mar 2007 - Newbie questions: Load-balanced Dovecot with NFS storage

Hi,

I experimented with Dovecot a while back and our site is now taking  
the plunge to switch to it from the UW IMAP server when we rejig the  
topology of our mail store and IMAP servers.

Currently we have a farm of 5 IMAP servers running the UW server.   
Each has locally attached disk used to store people's mailboxes.   
Each user is allocated to one of these five servers and can only  
access their mail via that server.  Our aim is to remove this  
restriction to provide resilience in the event of a server failing.

Our thoughts...

*  We are envisaging having a farm of N (to be determined) servers  
running
    Dovecot.  These will be closed systems: users do not have shell- 
access,
    and can only reach their mail via IMAP.

*  A single mailstore common to all N servers will be hosted on our  
NetApp
    filer and NFS-mounted on each server.  Each server would be running
    Solaris 10.  If we use all-new kit it will be x86-based, or we  
may have
    to redeploy some of our existing SPARC boxes to the new service.

*  We will be converting user mailboxes to be in Maildir format.

*  Our hardware load balancer, which supports persistent sessions,  
will be
    used to distribute users between the servers.

    This could either direct a user to any one of the N servers, or  
could be
    set to 'prefer' a given server for a particular user (only  
failing over
    to one of the others if their 'usual' server was unavailable).   
More on
    this in my questions below. :-)

I have been scouring the Dovecot list, archives and Wiki and think I  
have come up with a set of issues we need to be aware of and things  
we have to do in order to make this work reliably.

Please would you let me know if I have missed or misunderstood anything?

Issues To Watch For
-------------------
*  For timestamp integrity our servers will be synched with NTP to our
    local time server.

*  Because sessions could be on different servers memory mapping of  
index
    files doesn't work well with NFS, so set
        mmap_disable=yes
   in dovecot.conf.

*  Dovecot relies on the mtime timestamp of mailbox files so the
    NFS-mounted mailstore needs to be mounted with these options on each
    server:
        actimeo=0
    (Are there any other mount options we should use too?)

*  We will be using filesystem quotas on users mailboxes.  We understand
    that Dovecot's index files are best on no-quota filestores so  
will store
    these separately.

Questions
---------
Q1. Would it be better to store the index files on NFS-shared  
filestore and
     direct users to any of the IMAP server machines?  Or to store  
the index
     files on local disk and direct each user to their 'preferred' IMAP
     server machine?

Q2. Does Dovecot (or "something") clean out old index files that
haven't
     been accessed for a while?  Eg, when a user has temporarily come
     through on a different IMAP server to normal.  Or do the index  
files
     sit there untouched for evermore?

Q3. Storing the index files on the NetApp filer would give us the  
ability
     to share them between servers and grow their volume as need be,  
but at
     the cost of performance.  How big do the index files get?  Are they
     typically a few kBytes per message? Per mailbox? Per user? Or more?

Q4. We will be using Exim as the MTA, which can deliver direct to  
Maildir
     mailboxes.  However I understand that Dovecot's "deliver" LDA
adds the
     benefit of updating the index files as each message is  
delivered.  Is
     this a significant gain?  Or is there little difference in  
actual use?

Q5. We have around 20,000 mail accounts and will therefore be seeing  
lots
     of concurrent IMAP sessions, usually secure (SSL) ones.  I have  
seen
     mention that this can give rise to "Too many open files" errors  
under
     Solaris.  How do we avoid this when we are likely to have several
     thousand concurrent IMAP sessions per server machine?

With many thanks for your time and advice...

Cheers,
Mike Brudenell

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *

On Fri, Mar 02, 2007 at 04:12:10PM +0000, Mike Brudenell
wrote:
> *  Because sessions could be on different servers memory mapping of  
> index
>    files doesn't work well with NFS, so set
>        mmap_disable=yes
>   in dovecot.conf.
If your load balancer is set up to have persistent servers based upon
user criteria if some sort, you could actually store the indexes on
local drives on each machine.  Worst case scenario, if user flipped to
a different box in the cluster, Dovecot would have to rebuild its
index increasing CPU and I/O.  Best case, you see a performance gain
by using local store and reducing NFS traffic.
> *  Dovecot relies on the mtime timestamp of mailbox files so the
>    NFS-mounted mailstore needs to be mounted with these options on each
>    server:
>        actimeo=0
>    (Are there any other mount options we should use too?)
This is absolutely critical, yes.  Without this, all sorts of nasty
things can happen.  You can also use the "noac" attribute, at least
as of Solaris 8.
> *  We will be using filesystem quotas on users mailboxes.  We understand
>    that Dovecot's index files are best on no-quota filestores so  
> will store these separately.
The only issue here is dotlocking.  If your MTA or if Dovecot uses
dotlocks of any kind, you'll need to be sure those are also outside
of your quota-restricted filestore.  Otherwise, you are going to need
to be sure that the filesystem quota is never actually reached, and
that the limit is artificially set lower than the filesystem quota
in some other manner.
> Q1. Would it be better to store the index files on NFS-shared  
> filestore and
>     direct users to any of the IMAP server machines?  Or to store  
> the index
>     files on local disk and direct each user to their 'preferred'
IMAP
>     server machine?
Our plan is to store index files on local store and load balance to
persistent servers.  Sure, the persistent cache table expires over
time, but then again, the indexes get out of date over time anyway.
> Q2. Does Dovecot (or "something") clean out old index files that
haven't
>     been accessed for a while?  Eg, when a user has temporarily come
>     through on a different IMAP server to normal.  Or do the index  
> files
>     sit there untouched for evermore?
They sit untouched forever.  Feel free to remove them after they get to
be of certain age.
> Q3. Storing the index files on the NetApp filer would give us the  
> ability
>     to share them between servers and grow their volume as need be,  
> but at
>     the cost of performance.  How big do the index files get?  Are they
>     typically a few kBytes per message? Per mailbox? Per user? Or more?
I considered this, but my concern is reliability.  Dovecot's index
files seem a bit "delicate" in recent patches and I'm afraid of
possible issues of sharing them between servers, especially if
there are multiple IMAP sessions open on different servers.  I'm not
familiar with ultimate size at this point, but it depends on the
maximum size of the mailboxes and folders I suppose.
> Q4. We will be using Exim as the MTA, which can deliver direct to  
> Maildir
>     mailboxes.  However I understand that Dovecot's "deliver"
LDA
> adds the
>     benefit of updating the index files as each message is  
> delivered.  Is
>     this a significant gain?  Or is there little difference in  
> actual use?
No experience with the LDA component.
> Q5. We have around 20,000 mail accounts and will therefore be seeing  
> lots
>     of concurrent IMAP sessions, usually secure (SSL) ones.  I have  
> seen
>     mention that this can give rise to "Too many open files"
errors
> under
>     Solaris.  How do we avoid this when we are likely to have several
>     thousand concurrent IMAP sessions per server machine?
Yow.  Thousands of concurrent IMAP sessions *per* server?  All using
SSL?  With only 20,000 mail accounts?  Are you sure about that?  That
seems like an awfully high active-reader ratio given the low number of
accounts.  Still, if true, it is what it is and needs to be accomodated.

Obviously increasing the number of systems in your cluster is one way
to fight it.  I know there was a recent bug in Dovecot that was
causing file descriptor leaks, but if I recall it was fixed in a
recent patch.

Still, if you are going to have thousands of concurrent IMAP sessions,
I would consider making sure you have a good number of systems in
your cluster.

I'm not sure if you are using a webmail client, such as Squirrelmail, 
but if so you may also want to consider running an IMAP proxy server
to keep sessions open and persistent between page loads.

--
Dean Brooks
dean at iglou.com

Greetings -

I've got Dovecot 1.0rc24 up and running with both Maildirs and index  
files in the same user directory on NFS-mounted filestore.  Now I'm  
slowly edging things into a better, more efficient and NFS-friendly  
setup...

Currently I'm trying to move the index and control files off the NFS- 
mounted quota'd filestore and onto local disk without quotas.

Q1.  We are going to be running multiple load-balanced servers.  I know
      I can safely put the index files onto locally attached disk and  
they
      are rebuilt as and when needed.

      But is the same true for control files?  Or do they need to be
      preserved and made available to any server the user may come in
      through?  (This would imply having to have a different NFS-mounted
      filestore, as the NFS-mounted mailstore will have filestore quotas
      enabled, and I understand control files can be unhappy in this
      environment when the user runs out of quota.)


To break up large directories I want to create these in the form
     /var/dovecot/X/username/control
     /var/dovecot/X/username/index
where X is the first letter of the username.  So in dovecot.conf I have:

mail_location = maildir:/mailstore/%1u/%u/Maildir:INDEX=/var/dovecot/% 
1u/%u/index:CONTROL=/var/dovecot/%1u/%u/control

Q2.  An alternative would be to arrange the index and control diretories
      so they both hang off a single per-user area:
          /var/dovecot/X/username/control
          /var/dovecot/X/username/index
      However this again assumes I can safely delete files in the  
control
      directory tree if they aren't touched for a while.  Is this true,
      or are they 'valuable' and should be preserved for the lifetime of
      the account?

      (This is in part Q1 but phrased differently, I guess. :-)


I also hit some ownership/protection problems.  I'd originally  
assumed that I just had to create the top level
     /var/dovecot/{control,index}
directories and these could be owned root:root with mode 755.  But  
Dovecot fails to create these and logs entries such as:

     dovecot: Mar 07 12:00:47 Error: IMAP(pmb1)[2835]:
         mkdir(/var/dovecot/index/p/pmb1/.INBOX) failed: Permission  
denied

Experimentation suggests that the instance of Dovecot wanting to  
create the directories is running under the user's uid/gid (me, in  
this case).  Hence I now believe:

1.  I need to manually pre-create the next level down directories
         /var/dovecot/{control,index}/[a-z]

2.  Set these to be owned root:root with mode rwxrwxrwt

3.  I can periodically clean out 'old' files in the index tree.
     (Not sure about the control tree: hence Q1 and Q2 above.)


This seems to work OK... I just wanted to check I hadn't missed  
something simpler and blindingly obvious?

Cheers,
Mike B-)

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *