On Jul 16, 2012, at 8:58 AM, Robert Blayzor wrote:
> In reading the Dovecot NFS WIKI entry:
>
> http://wiki2.dovecot.org/NFS
>
> It mentions a FreeBSD NFS client caching bug. I believe that bug only
exists pre-8.x ? It seems that a patch in the PR notes was MFC'd and made
it into 8.x at some point. (at least I checked the source in 8.3 and it was
there). So I assume that FreeBSD 8.x mitigates the issue? Maybe not?
>
>
> So our deployment is multi server Dovecot/Exim running on FreeBSD servers,
frontended by a layer3 load balancing switch and a NetApp NFSv3 datastore.
Our deployment is similar. Dovecot/Exim on Oracle Linux 6.2 (RHEL derivative),
layer3 load balancing (pacemaker/ldirectord managing LVS, "sticky"
sessions per client IP) and NetApp NFSv3 datastore (maildir).
>
> The only real concern we have right now is race conditions and file write
contention on the Dovecot index files. Our original plan was to use Dovecot
LMTP on the servers to deliver to the inbox, but that doesn't seem to be
recommended as we'll have a number of servers all acting as primary MX for
domains and delivering to the same set of mailboxes. So are the Dovecot dotlock
file locking mechanisms not enough for multiple servers trying to write/access
index files in Maildirs?
I can't speak to FreeBSD but we have had no problems using the NetApp NFS
lock manager (lock_method = fcntl) instead of dotlock. Previously we were using
dotlock and any skew in clocks on the Dovecot servers would cause higher CPU
usage and tons of extra logging. If it works for you, try to use the NetApp NFS
lock manager instead of dotlock.
It sounds like you have a bunch of hosts running both Exim and Dovecot and they
are all your MX hosts? If so, it sounds like you could use Dovecot's LDA
binary or LMTP. Since you are running Dovecot also on your Exim hosts and thus
don't need to go over the network to a different host to see the
mailstore/deliver the message, I would stick with LDA but both options will
work.
>
> If we have to deliver straight to the ~/Maildir/new out of Exim, that's
fine, also. We're just wondering if there is a added benefit of actually
using LMTP. (other than making sure the index are kept the most recent).
>
I believe the latest word from Timo is that the benefit of index updating from
LDA/LMTP delivery for maildir is minimal, especially if your indexes are on NFS.
We do use LDA for delivery with indexes on NFS, and have not had problems (~7000
user site). The important settings are:
# Don't use mmap() at all. This is required if you store indexes to shared
# filesystems (NFS or clustered filesystem).
#mmap_disable = no
mmap_disable = yes
# When to use fsync() or fdatasync() calls:
# optimized (default): Whenever necessary to avoid losing important data
# always: Useful with e.g. NFS when write()s are delayed
# never: Never use it (best performance, but crashes can lose data)
#mail_fsync = optimized
mail_fsync = always
# Mail storage exists in NFS. Set this to yes to make Dovecot flush NFS caches
# whenever needed. If you're using only a single mail server this isn't
needed.
#mail_nfs_storage = no
mail_nfs_storage = yes
# Mail index files also exist in NFS. Setting this to yes requires
# mmap_disable=yes and fsync_disable=no.
#mail_nfs_index = no
mail_nfs_index = yes
> The other concern we have is client mailbox access. While we can make
sessions "sticky" from one IP so they hit the same server, it
doesn't help with the person that checks their email via POP from their PC
while having their mobile phone access it via IMAP. (obviously from different IP
addresses). So I guess there is some possibly index file write contention there
as well. (though hopefully this should be rare).
We make sessions "sticky" and have the same "different client IP,
same username" pattern. We haven't had problems with this setup since
switching away from dotlock to NFS locks.
>
> I'm curious as to know others experiences with a similar deployment.
>
I should mention we are hoping to change this deployment soon. As others have
said, you would probably be better served at least with Dovecot Director (to
ensure client access and Exim deliveries using LMTP for user X always hits
server Y) and possibly mdbox instead of maildir. We have been testing Director
and I feel good enough about it to move indexes off of NFS onto local disk and
setting a scheduled task to update indexes on every server at some reasonable
interval so a server failure doesn't require a complete index rebuild when
users tied to the failed server get sent to a different one. Migrating from
maildir to mdbox is on my to-investigate list once Director is implemented but
without having tested a migration there is a lot to like in terms of Alternate
Storage and reduced IOPS v Maildir. You just have to *really* trust those index
files because they are the only place flags/keywords are stored.
> TIA
>
> --
> Robert Blayzor
> INOC, LLC
> rblayzor at inoc.net
> http://www.inoc.net/~rblayzor/
>
>
>
HTH,
David Warden