dovecot - Oct 2009 - NFS random redirects

Hello,

Dovecot documentation states that the "random redirects to multiple
servers"
NFS solution is to be avoided and I'm investigation the actual risks of it
and
a way to put it to test.

I'm running dovecot-1.2.6 with Maildir (indexes, mailboxes and control files
are all on NFS) and I'm using procmail instead of deliver as the LDA.

  1. Documentation says : "Dovecot locks the maildir while doing
modifications
     to it or while looking for new messages in it" and then mention the
     "dovecot-uidlist.lock" dotlock file.

      a) if that file fcntl'ed in addition (i.e. dotlocked + fcntled or just
dotlocked) ?

      b) is that file THE way to lock the Maildir mentionned above or is it
        just something else used only for updating that particular file (i.e. is
        Maildir locked in some way + dovecot-uidlist.lock created or just
        dovecot-uidlist.lock created) ?

  2. Documentation says : "NFS caching is a big problem when multiple
computers
     are accessing the same mailbox simultaneously"

     I guess it's because of the dotlocks files and not fcntl locking ? Is
     there anything else than dovecot-uidlist which is dotlocked ? If yes, is it
     dotlocked only or dotlocked and fcntled ?

  3. Documentation says : "Dovecot v1.1 flushes NFS caches when needed if
you set mail_nfs_storage=yes"

     How can a program flush the NFS caches ? By which (system) call ?

  4. Documentation says : "Besides the NFS cache problems described above,
     mailbox contents can't be cached as well in the memory either."

    Is it about in-memory indexes or part of indexes loaded into memory ?

  5. How can I torture-test concurrent access to the same mailbox through 2
     dovecot servers ? I don't see imaptest.c anymore on
http://dovecot.org/tools
     but I see 2 nfs_test ? Is there some command line options I should use ?

  6. when and why can a random redirect to multiple server can cause mailbox
     corruption ? On flags only or on content as well ?

Thanks

-- 
Thomas Hummel 	    | Institut Pasteur
<hummel at pasteur.fr> | P?le informatique - syst?mes et r?seau

On Oct 19, 2009, at 9:35 AM, Thomas Hummel wrote:
> Dovecot documentation states that the "random redirects to multiple  
> servers"
> NFS solution is to be avoided and I'm investigation the actual risks  
> of it and
> a way to put it to test.
If you do it, you'll most likely see some random index related errors.
>  1. Documentation says : "Dovecot locks the maildir while doing  
> modifications
>     to it or while looking for new messages in it" and then mention  
> the
>     "dovecot-uidlist.lock" dotlock file.
>
>      a) if that file fcntl'ed in addition (i.e. dotlocked + fcntled  
> or just dotlocked) ?
No fcntl.
>      b) is that file THE way to lock the Maildir mentionned above or  
> is it
>        just something else used only for updating that particular  
> file (i.e. is
>        Maildir locked in some way + dovecot-uidlist.lock created or  
> just
>        dovecot-uidlist.lock created) ?
Maildir is then locked for Dovecot against doing any modifications.  
Procmail will ignore it, and that's fine.
>  2. Documentation says : "NFS caching is a big problem when multiple  
> computers
>     are accessing the same mailbox simultaneously"
>
>     I guess it's because of the dotlocks files and not fcntl locking ?
No, it's because of NFS caching.. Like if Dovecot writes to  
dovecot.index.log first and then dovecot.index, it expects that it's  
not possible that a later read sees the dovecot.index change, but not  
the dovecot.index.log change. But because of NFS caching this is  
possible, and "corrupted index files" errors will occur. There are  
many other similar issues.
>  3. Documentation says : "Dovecot v1.1 flushes NFS caches when  
> needed if you set mail_nfs_storage=yes"
>
>     How can a program flush the NFS caches ? By which (system) call ?
That's the main problem. There is no reliable way to flush NFS  
caches.. So I've added some highly OS-specific code that works most of  
the time, but not perfectly. It works best with Linux.
>  4. Documentation says : "Besides the NFS cache problems described  
> above,
>     mailbox contents can't be cached as well in the memory
either."
>
>    Is it about in-memory indexes or part of indexes loaded into  
> memory ?
I was thinking about OS's filesystem caching in general, so all mail  
files and such.
>  5. How can I torture-test concurrent access to the same mailbox  
> through 2
>     dovecot servers ? I don't see imaptest.c anymore on
http://dovecot.org/tools
>     but I see 2 nfs_test ? Is there some command line options I  
> should use ?
imaptest exists now in http://imapwiki.org/ImapTest
>  6. when and why can a random redirect to multiple server can cause  
> mailbox
>     corruption ?
Pretty much randomly..
> On flags only or on content as well ?
Actual mail content should be safe. Unless you just happen to get such  
a cache file corruption that Dovecot doesn't notice it and sends some  
broken headers to IMAP client.

The long term fix for this is
http://www.dovecot.org/list/dovecot/2009-August/041983.html