dovecot - Mar 2010 - corrupted mbox

Heya,

we have two IMAP (UW-imap and dovecot) servers accessing the same mbox files for
our users (don't ask why) and sometimes the UW-imap corrupts the mbox file.
The mbox looks just like normal mbox except it does not have the proper
'From ' beginning of the file (usually there are some random tidbits in
the begining of the file, which can be easily stripped to recover the working
mbox).

The problem is that if we use dovecot for POP3 server, the dovecot logs the mbox
corruption just fine:
POP3(username): Couldn't init INBOX: Mailbox isn't a valid mbox file

But if we use IMAP for the same mailbox, dovecot opens the file and does not
report any corruption, the user on the other hand sees just empty mailbox.

So I was wondering if there are any possibilities to report mbox corruption with
IMAP as well?

thanks in advance,
regards, Jernej

On Fri, 2010-03-26 at 13:27 +0100, Jernej Porenta wrote:
> Heya,
> 
> we have two IMAP (UW-imap and dovecot) servers accessing the same mbox
> files for our users (don't ask why) and sometimes the UW-imap corrupts
> the mbox file. The mbox looks just like normal mbox except it does not
> have the proper 'From ' beginning of the file (usually there are
some
> random tidbits in the begining of the file, which can be easily
> stripped to recover the working mbox).
The corruption shouldn't be happening if both use the same locking
mechanisms. I think UW-IMAP uses dotlock + flock. See if changing
Dovecot to use those avoids the corruption?
> The problem is that if we use dovecot for POP3 server, the dovecot
> logs the mbox corruption just fine:
> POP3(username): Couldn't init INBOX: Mailbox isn't a valid mbox
file
> 
> But if we use IMAP for the same mailbox, dovecot opens the file and
> does not report any corruption, the user on the other hand sees just
> empty mailbox.
What Dovecot version? The IMAP code is pretty much the same as POP3, so
I don't really see why IMAP wouldn't be noticing it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100326/6ce5b3b1/attachment-0002.bin>

On Mar 26, 2010, at 1:39 PM, Timo Sirainen wrote:
> On Fri, 2010-03-26 at 13:27 +0100, Jernej Porenta wrote:
>> Heya,
>> 
>> we have two IMAP (UW-imap and dovecot) servers accessing the same mbox
>> files for our users (don't ask why) and sometimes the UW-imap
corrupts
>> the mbox file. The mbox looks just like normal mbox except it does not
>> have the proper 'From ' beginning of the file (usually there
are some
>> random tidbits in the begining of the file, which can be easily
>> stripped to recover the working mbox).
> 
> The corruption shouldn't be happening if both use the same locking
> mechanisms. I think UW-IMAP uses dotlock + flock. See if changing
> Dovecot to use those avoids the corruption?
The corruptions were happening even before we started using dovecot, so I
believe the problem exists in old uw-imap release that we (have to) use, because
of our custom made patches. We will be eliminating uw-imap in April, but we have
to take care of the corrupted mboxes before that ;)
>> The problem is that if we use dovecot for POP3 server, the dovecot
>> logs the mbox corruption just fine:
>> POP3(username): Couldn't init INBOX: Mailbox isn't a valid mbox
file
>> 
>> But if we use IMAP for the same mailbox, dovecot opens the file and
>> does not report any corruption, the user on the other hand sees just
>> empty mailbox.
> 
> What Dovecot version? The IMAP code is pretty much the same as POP3, so
> I don't really see why IMAP wouldn't be noticing it.
Dovecot version is 1.2.11 with almost default settings.

Here are the logs for the user with corrupted mbox, who tried POP3 first and
afterwards IMAP:
Mar 19 10:34:54 machine dovecot: pop3-login: Login: user=<username>,
method=PLAIN, rip=X.X.X.Z, lip=X.X.X.W, secured
Mar 19 10:34:54 machine dovecot: POP3(username): Effective uid=15657, gid=3400,
home=/mnt/./39/username
Mar 19 10:34:54 machine dovecot: POP3(username): mbox:
data=~/:INBOX=~/.mailbox:INDEX=/mnt/dovecot-index/l/username
Mar 19 10:34:54 machine dovecot: POP3(username): fs: root=/mnt/./39/username,
index=/mnt/dovecot-index/l/username, control=,
inbox=/mnt/./39/username/.mailbox/
Mar 19 10:34:54 machine dovecot: POP3(username): Couldn't init INBOX:
Mailbox isn't a valid mbox file
Mar 19 10:34:54 machine dovecot: POP3(username): Mailbox init failed top=0/0,
retr=0/0, del=0/0, size=0
Mar 19 10:43:18 machine dovecot: imap-login: Login: user=<username>,
method=PLAIN, rip=X.X.X.Y, lip=X.X.X.W
Mar 19 10:43:18 machine dovecot: IMAP(username): Loading modules from directory:
/opt/dovecot/lib/dovecot/imap
Mar 19 10:43:18 machine dovecot: IMAP(username): Module loaded:
/opt/dovecot/lib/dovecot/imap/lib10_quota_plugin.so
Mar 19 10:43:18 machine dovecot: IMAP(username): Module loaded:
/opt/dovecot/lib/dovecot/imap/lib11_imap_quota_plugin.so
Mar 19 10:43:18 machine dovecot: IMAP(username): Effective uid=15657, gid=3400,
home=/mnt/./39/username
Mar 19 10:43:18 machine dovecot: IMAP(username): Quota root: name= backend=fs
argsMar 19 10:43:18 machine dovecot: IMAP(username): mbox:
data=~/:INBOX=~/.mailbox:INDEX=/mnt/dovecot-index/l/username
Mar 19 10:43:18 machine dovecot: IMAP(username): fs: root=/mnt/./39/username,
index=/mnt/dovecot-index/l/username, control=, inbox=/mnt/./39/username/.mailbox
Mar 19 10:43:18 machine dovecot: IMAP(username): fs quota add storage dir =
/mnt/./39/username
Mar 19 10:43:18 machine dovecot: IMAP(username): fs quota block device =
/dev/dsk/c7t600D0230FFFFFFFF01EBFD22D8275C00d0s2
Mar 19 10:43:18 machine dovecot: IMAP(username): fs quota mount point = /mnt/39
Mar 19 10:43:18 machine dovecot: IMAP(username): Namespace : Using permissions
from /mnt/./39/username: mode=0711 gid=-1
Mar 19 10:43:18 machine dovecot: IMAP(username): Disconnected: Logged mnt
bytes=254/909

I would be happy if IMAP would drop a line about the corrupted mbox... :)

thanks in advance, regards, J.