dovecot - Jun 2008 - Dovecot index, NFS, and multiple architectures

Hi all,

I crawled through the archives for a bit but didn't see anything 
helpful, so I apologize if this has already been addressed. We've been 
dying to move from Courier to Dovecot across our whole infrastructure 
for quite some time, but until recently our setup wasn't possible until 
this happened:

"Dovecot allows mailboxes and their indexes to be modified by multiple 
computers at the same time, while still performing well. This means that 
Dovecot works with NFS and clustered filesystems."

Now that NFS is officially supported, we figured why not make the 
switch. All of our mail touches NFS in some way, so we need to check on 
the stability before completely migrating. In test trials we're having 
issues with the NFS'd index becoming corrupted. Here's the setup:

* Linux workstations running Fedora 8/9 i386 and a locally called 
Dovecot 1.0.14
* NFS'd homedir with Maildir setup
* NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14

Now this setup is just a test example and not exactly what we'll be 
running in production, but it tipped up the problem either way. Since 
the index is shared by both the Linux i386 machine and the sparc64 
Solaris machine, if mail is accessed from both, lets say with Pine for 
example, the index becomes corrupted and breaks. As long as only one 
architecture only ever touches it there are no issues.

I'm assuming this is an endian issue, which would make the most sense. 
Is there a way around this with flags or server options? Is this 
something that has maybe been addressed in 1.1.0?

Unfortunately, we are not going to be able to move to Dovecot across all 
of our systems until this is no longer an issue. We run a lot of mixed 
environments that have everything from Linux i386/x86_64 to Solaris 9 64 
to Windows. If a user accesses IMAP from a Windows box, then logs into 
the front end which is a Linux x86_64 box and runs Pine,  and all of 
this is on a Solaris sparc NFS system, we're going to have serious 
issues with the index. ;)

Any help would be appreciated.
Thanks,
-Dave

-- 
===============================David Halik
System Administrator
OIT-CSS Rutgers University
dhalik at jla.rutgers.edu
http://www.digitalruin.net
================================

On 6/18/2008, David Halik (dhalik at jla.rutgers.edu)
wrote:
> * Linux workstations running Fedora 8/9 i386 and a locally called Dovecot
1.0.14
> * NFS'd homedir with Maildir setup
> * NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14 
NFS is only fully supported on 1.1+. This is why you're having trouble.

1.1rc10 is the latest, and the stable release is imminent, so you should 
have no trouble moving to it.

-- 

Best regards,

Charles

On Jun 18, 2008, at 9:12 PM, David Halik wrote:
> Now this setup is just a test example and not exactly what we'll be  
> running in production, but it tipped up the problem either way.  
> Since the index is shared by both the Linux i386 machine and the  
> sparc64 Solaris machine, if mail is accessed from both, lets say  
> with Pine for example, the index becomes corrupted and breaks. As  
> long as only one architecture only ever touches it there are no  
> issues.
It should only log "architecture changed" and rebuild the index. I did
some fixed related to this at some point though, might have been only  
for v1.1.
> I'm assuming this is an endian issue, which would make the most  
> sense. Is there a way around this with flags or server options? Is  
> this something that has maybe been addressed in 1.1.0?
No. I once thought about making it endianess-independent, but rejected  
the idea soon after I figured out that it would unnecessarily slow  
down nearly-100% of users and the code would become really ugly.

Although this reminds me .. dbox does rely on indexes more than mbox/ 
maildir. Perhaps there should be code that attempts to preserve and  
convert the important data (flags, keywords) from different endianess  
indexes. But this still wouldn't make it possible to quickly switch  
between architectures.

One thing you could do is to specify different index files paths for  
your installations. For example for sparc64:

mail_location = maildir:~/Maildir:INDEX=~/Maildir/sparc64

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080618/92d4f742/attachment-0002.bin>

I just reproduced the environment and the index corrupted immediately 
across NFS because of the endian issue.

Jun 25 11:53:34 host IMAP(user): : Rebuilding index file 
/dovecot-index/index/user/.INBOX/dovecot.index: CPU architecture changed
Jun 25 11:53:35 host IMAP(user): : Corrupted index cache file 
/dovecot-index/index/user/.INBOX/dovecot.index.cache: field header 
points outside file

This was starting from a clean index, first opening pine on the NFS 
Solaris 9 sparc machine, and then at the same time opening pine on my 
Fedora 9 i386 workstation.

I'm going to try the idea of splitting the indexes into two different 
architectures, but I'm worried that this will not be feasible when we 
try to scale to our 80,000 users.

By the way, I don't think this is related to the corruption, but we also 
have tons of these in the logs:

Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is 
different than current time (1214409234 vs 1214409152): 
/dovecot-index/control/user/.INBOX/dovecot-uidlist
Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is 
different than current time (1214409235 vs 1214409152): 
/dovecot-index/control/user/.INBOX/dovecot-uidlist


Timo Sirainen wrote:
> On Jun 18, 2008, at 9:12 PM, David Halik wrote:
>
>> Now this setup is just a test example and not exactly what we'll be
>> running in production, but it tipped up the problem either way. Since 
>> the index is shared by both the Linux i386 machine and the sparc64 
>> Solaris machine, if mail is accessed from both, lets say with Pine 
>> for example, the index becomes corrupted and breaks. As long as only 
>> one architecture only ever touches it there are no issues.
>
> It should only log "architecture changed" and rebuild the index.
I did
> some fixed related to this at some point though, might have been only 
> for v1.1.
>
>> I'm assuming this is an endian issue, which would make the most 
>> sense. Is there a way around this with flags or server options? Is 
>> this something that has maybe been addressed in 1.1.0?
>
> No. I once thought about making it endianess-independent, but rejected 
> the idea soon after I figured out that it would unnecessarily slow 
> down nearly-100% of users and the code would become really ugly.
>
> Although this reminds me .. dbox does rely on indexes more than 
> mbox/maildir. Perhaps there should be code that attempts to preserve 
> and convert the important data (flags, keywords) from different 
> endianess indexes. But this still wouldn't make it possible to quickly 
> switch between architectures.
>
> One thing you could do is to specify different index files paths for 
> your installations. For example for sparc64:
>
> mail_location = maildir:~/Maildir:INDEX=~/Maildir/sparc64
>

-- 
===============================David Halik
System Administrator
OIT-CSS Rutgers University
dhalik at jla.rutgers.edu
================================