dovecot - Mar 2008 - UID inserted in the middle of mailbox errors

Hi,

We are still getting, on a daily basis, users who cannot move messages
to Trash or expunge Trash due to these errors:

   Mar 28 10:43:57 tm2 dovecot: [ID 107833 mail.error] IMAP(kss021):
   mbox sync: UID inserted in the middle of mailbox
   /home/kss021/Mail/Trash (3137 > 536, seq=2, idx_msgs=76)

Our users have no other access to their mailbox other than Dovecot,
and we are using mbox format and 1.1rc3.

Deleting the indexes fixes the problem, so it seems to be a problem
with index corruption and not something inherently wrong with the
mailbox.  That is, once I delete the index files, the problem does not
come back immediately.

I've seen others on the list with the same problem, but I wasn't able
to find a workaround.  It's happening more and more frequently.

I guess what I don't understand is why Dovecot cant just mark the index
invalid and purge it instead of failing over and over.  I thought the
whole point behind Dovecot's indexing is that it was able to detect
index inconsistencies and mark the index invalid.

Any help is much appreciated.  Even a quick stopgap fix would be
a lifesaver, short of a cronjob that just purges everyone's index
files every 2 hours (which sort of defeats the point of using Dovecot).

--
Dean Brooks
dean at iglou.com

On Fri, 28 Mar 2008 10:53:49 -0400 Dean Brooks wrote:
> Hi,
> 
> We are still getting, on a daily basis, users who cannot move messages
> to Trash or expunge Trash due to these errors:
> 
>    Mar 28 10:43:57 tm2 dovecot: [ID 107833 mail.error] IMAP(kss021):
>    mbox sync: UID inserted in the middle of mailbox
>    /home/kss021/Mail/Trash (3137 > 536, seq=2, idx_msgs=76)
> 
> Our users have no other access to their mailbox other than Dovecot,
> and we are using mbox format and 1.1rc3.
With maildir format too 

Mar 27 10:19:26 mailbox dovecot: imap(exim4000): Maildir
/var/Mail/Boxes/e/exim4000/.Trash sync: UID inserted in the middle of mailbox
(105265 > 105264, file =
1206604287.H478110P28054.mailbox.tu-berlin.de,S=6185:2,b)

On Fri, Mar 28, 2008 at 10:53:49AM -0400, Dean Brooks wrote:
> We are still getting, on a daily basis, users who cannot move messages
> to Trash or expunge Trash due to these errors:
> 
>    Mar 28 10:43:57 tm2 dovecot: [ID 107833 mail.error] IMAP(kss021):
>    mbox sync: UID inserted in the middle of mailbox
>    /home/kss021/Mail/Trash (3137 > 536, seq=2, idx_msgs=76)
> 
> Our users have no other access to their mailbox other than Dovecot,
> and we are using mbox format and 1.1rc3.
> 
> Deleting the indexes fixes the problem, so it seems to be a problem
> with index corruption and not something inherently wrong with the
> mailbox.  That is, once I delete the index files, the problem does not
> come back immediately.
> 
> I've seen others on the list with the same problem, but I wasn't
able
> to find a workaround.  It's happening more and more frequently.
> 
> I guess what I don't understand is why Dovecot cant just mark the index
> invalid and purge it instead of failing over and over.  I thought the
> whole point behind Dovecot's indexing is that it was able to detect
> index inconsistencies and mark the index invalid.
For what it's worth, I set "mbox_lazy_writes = no" in dovecot.conf
and the problem hasn't happened all weekend.  Previously, it was
happening hourly.

It seems that there may be some lingering bugs in the lazy_write logic.
If anyone is running into this problem on a regular basis, I would
try disabling lazy writes and see if that helps.

I'm not sure why everyone isn't having this problem given how such
a stock vanilla configuration we're using, but maybe it's just related
to server volume.

--
Dean Brooks
dean at iglou.com

On Fri, 2008-03-28 at 10:53 -0400, Dean Brooks wrote:
>    Mar 28 10:43:57 tm2 dovecot: [ID 107833 mail.error] IMAP(kss021):
>    mbox sync: UID inserted in the middle of mailbox
>    /home/kss021/Mail/Trash (3137 > 536, seq=2, idx_msgs=76)
..
> I guess what I don't understand is why Dovecot cant just mark the index
> invalid and purge it instead of failing over and over.  I thought the
> whole point behind Dovecot's indexing is that it was able to detect
> index inconsistencies and mark the index invalid.
It wasn't actually failing at all from user's point of view. It just
kept logging the same error over and over again because it always
delayed writing fixes to the mbox file.

I changed the code to now always write pending changes to mbox if
syncing had errors. This still causes errors to be written twice, but
that can't be fixed easily.
http://hg.dovecot.org/dovecot-1.1/rev/591145eaaf87

As for why these happen in the first place when copying, I've really no
idea.. I tried to reproduce this with imaptest, but couldn't. Does
imaptest show any errors with you? http://imapwiki.org/ImapTest

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080504/9704a984/attachment-0002.bin>