dovecot - Jan 2012 - force-resync fails to recover all messages in mdbox

To my understanding, when using mdbox, doveadm force-resync
should be able to recover all the messages from the storage files
alone, though of course losing all metadata except the initial
delivery folder.

However, this does not seem to be the case. For me, force-resync
creates only partial indices that lose messages. The message
contents are of course still in the storage files, but dovecot
just doesn't seem to be aware of some of them after recreating
the indices.

Here is an example. I created a test mdbox by syncing a mailing
list folder from a mbox location:

$ dsync -m haskell-cafe backup mdbox:~/dbox

Then I switched the location to the new mdbox:

$ /usr/sbin/dovecot -n
# 2.0.15: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-0.bpo.1-amd64 x86_64 Debian wheezy/sid
mail_fsync = never
mail_location = mdbox:~/dbox
mail_plugins = zlib
passdb {
   driver = pam
}
plugin {
   sieve = ~/etc/sieve/dovecot.sieve
   sieve_dir = ~/etc/sieve
   zlib_save = bz2
   zlib_save_level = 9
}
protocols = " imap"
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
   driver = passwd
}
protocol lda {
   mail_plugins = zlib sieve
}

Then I checked the number of messages we had in the new location:

$ doveadm search all | wc
   93236  186472 3625098

This was indeed the correct number, so the sync was fine:

$ grep -c '^From .*..:..:.. ....$' ../mbox/haskell-cafe
93237

Then I removed all the indices and rebuilt them:

$ mkdir bak
$ mv mailboxes bak
$ mv storage/dovecot.map.index* bak/
$ doveadm force-resync inbox
doveadm(la): Warning: mdbox /home/la/dbox/storage: rebuilding indexes

And what have we now:

$ doveadm search all | wc
   43864   87728 1699590

Somehow dovecot lost over half of the messages!

This is really worrisome. It should always be possible to extract
the plain mail content out from the storage, but currently it
only _looks_ like it is possible: you get a problem in the
indices, rebuild them, and then later migrate somewhere else. If
you don't notice that you are missing mails, you may then
eventually destroy the original mdbox storage files, thinking
that their content is now safely elsewhere, when it really isn't.


Lauri

On 31.1.2012, at 17.48, Lauri Alanko wrote:
> $ doveadm search all | wc
>  93236  186472 3625098
..
> Then I removed all the indices and rebuilt them:
> 
> $ doveadm search all | wc
>  43864   87728 1699590
> 
> Somehow dovecot lost over half of the messages!
There may be a bug, and I just yesterday noticed something weird in the
rebuilding code. I'll have to look into that. But anyway, "search
all" isn't the proper way to test this. Try instead with:

doveadm fetch guid all | sort | uniq | wc

When you removed indexes Dovecot no longer knew about copies of messages.