dovecot - Jan 2012 - Problem with huge IMAP Archive after Courier migration

I have a mail server running Debian 6.0 with Courier IMAP to store 
project related mail.
Currently the maildir of the archive (one user) contains about 37GB of data.
Our staff is acessing the archive via Outlook 2007 where they drag their 
Exchange inbox or sent files to it.
The problem with courier is that is sometimes mixes up headers with 
message bodies, so I wanted to migrate to dovecot.

I tried this on my proxy running Debian 7.0 with some test data and this 
worked fine (OK, spent some hours to get the config files done - Dovecot 
without authentication).
Dovecot version here is 2.0.15.

Tried it with our productive system today, but got Dovecot 1.2.15 
installed on Debian 6.0
Config files and parameters I took from my test system were not 
compatible and I didn't get it to work.
So I forced to install the Debisn 7.0 packages with 2.0.15 and finally 
got the server running, I also restarted the whole machine to empty caches.
But the problem I got was that in the huge folder hierarchy the 
downloaded headers in the individual folders disappeared, some folders 
showed a few very old messages, some none. Also some subfolders disappeared.
I checked this with Outlook and Thunderbird. The difference was, that 
Thunderbird shows more messages (but not all) than Outlook in some 
folders, but also none in some others. Outlook brought up a message in 
some cases, that the connection timed out, although I set the timeout to 
60s.

After being frustrated uninstalled dovecot, went back to Courier and 
folder contents are displayed correctly again.

Anyone a clue what's wrong here?

Finally some config information:
proxy-server:~# dovecot -n
# 2.0.15: /etc/dovecot/dovecot.conf
# OS: Linux 3.1.0-1-686-pae i686 Debian wheezy/sid
auth_debug_passwords = yes
auth_mechanisms = plain login
disable_plaintext_auth = no
namespace {
   inbox = yes
   location    prefix = INBOX.
   separator = .
   type = private
}
passdb {
   driver = pam
}
plugin {
   sieve = ~/.dovecot.sieve
   sieve_dir = ~/sieve
}
protocols = imap
ssl = no
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
   driver = passwd
}

On 1/2/2012 10:17 AM, Preacher wrote:
...
> So I forced to install the Debisn 7.0 packages with 2.0.15 and finally
> got the server running, I also restarted the whole machine to empty caches.
> But the problem I got was that in the huge folder hierarchy the
> downloaded headers in the individual folders disappeared, some folders
> showed a few very old messages, some none. Also some subfolders
> disappeared.
> I checked this with Outlook and Thunderbird. The difference was, that
> Thunderbird shows more messages (but not all) than Outlook in some
> folders, but also none in some others. Outlook brought up a message in
> some cases, that the connection timed out, although I set the timeout to
> 60s.
...
> Anyone a clue what's wrong here?
Absolutely.  What's wrong is a lack of planning, self education, and
patience on the part of the admin.

Dovecot gets its speed from its indexes.  How long do you think it takes
Dovecot to index 37GB of maildir messages, many thousands per directory,
hundreds of directories, millions of files total?  Until those indexes
are built you will not see a complete folder tree and all kinds of stuff
will be missing.

For your education:  Dovecot indexes every message and these indexes are
the key to its speed.  Normally indexing occurs during delivery when
using deliver or lmtp, so the index updates are small and incremental,
keeping performance high.  You tried to do this and expected Dovecot to
instantly process it all:

http://www.youtube.com/watch?v=THVz5aweqYU

If you don't know, that's a coal train car being dumped.  100 tons of
coal in a few seconds.  Visuals are always good teaching tools.  I think
this drives the point home rather well.

-- 
Stan

On Mon, 2012-01-02 at 17:17 +0100, Preacher wrote:
> So I forced to install the Debisn 7.0 packages with 2.0.15 and finally 
> got the server running, I also restarted the whole machine to empty caches.
> But the problem I got was that in the huge folder hierarchy the 
> downloaded headers in the individual folders disappeared, some folders 
> showed a few very old messages, some none. Also some subfolders
disappeared.
> I checked this with Outlook and Thunderbird. The difference was, that 
> Thunderbird shows more messages (but not all) than Outlook in some 
> folders, but also none in some others. Outlook brought up a message in 
> some cases, that the connection timed out, although I set the timeout to 
> 60s.
Did you run the Courier migration script?
http://wiki2.dovecot.org/Migration/Courier

Also explicitly setting mail_location would be a good idea.

Yes i did, followed this guide you mentioned, it said that it found the 
3 mailboxes I have set-up in total, conversion  took only a few moments.
I guess the mail location was automaticall set correctly as the folder 
hierachy was displayed correctly

Timo Sirainen schrieb:
> On Mon, 2012-01-02 at 17:17 +0100, Preacher wrote:
>> So I forced to install the Debisn 7.0 packages with 2.0.15 and finally
>> got the server running, I also restarted the whole machine to empty
caches.
>> But the problem I got was that in the huge folder hierarchy the
>> downloaded headers in the individual folders disappeared, some folders
>> showed a few very old messages, some none. Also some subfolders
disappeared.
>> I checked this with Outlook and Thunderbird. The difference was, that
>> Thunderbird shows more messages (but not all) than Outlook in some
>> folders, but also none in some others. Outlook brought up a message in
>> some cases, that the connection timed out, although I set the timeout
to
>> 60s.
> Did you run the Courier migration script?
> http://wiki2.dovecot.org/Migration/Courier
>
> Also explicitly setting mail_location would be a good idea.
>
>
>