dovecot - Dec 2006 - Huge IMAP folder takes forever

Hi!
Just updated to 1.0 rc15, but the problem remains. I think it could be 
client-dependent to some extent.

I have only IMAPS access and Maildir. I have one mailbox which contains 
some 10,200 mails. That mailbox takes forever (have waited 30 minutes 
and then aborted) to open with Thunderbird 1.5.0.7. Other mailboxes 
(like dovecot :-), only 6984 mails) opens quickly.

When Thunderbird tries to open this folder, the imap process on the 
server uses roughly all cpu resources. There isn't much disk I/O, at 
least as far as I can tell by listening at the server.

The dovecot is upgraded from 0.99.something. Should I remove index files 
first, and in that case which? Is there something cached in Thunderbird?

Here's the directory listing for the troublesome folder:

-rw-------   1 peter  peter       28 Jan  7  2006 .customflags
-rw-------   1 peter  peter    49248 Feb  7  2006 .imap.index
-rw-------   1 peter  peter   420998 Feb  7  2006 .imap.index.data
-rw-------   1 peter  peter    10256 Feb  7  2006 .imap.index.log
-rw-------   1 peter  peter    49200 Feb  7  2006 .imap.index.tree
drwx------   2 peter  peter   686592 Dec 10 15:06 cur
-rw-------   1 peter  peter       64 Jun  6  2006 dovecot-keywords
-rw-------   1 peter  peter   619832 Dec 10 15:04 dovecot-uidlist
-rw-------   1 peter  peter   162992 Dec 10 15:21 dovecot.index
-rw-------   1 peter  peter  8826880 Dec  9 15:36 dovecot.index.cache
-rw-------   1 peter  peter    33776 Dec 10 15:22 dovecot.index.log
-rw-------   1 peter  peter   193660 Dec  3 12:50 dovecot.index.log.2
drwx------   2 peter  peter      512 Dec 10 15:06 new
drwx------   2 peter  peter      512 Dec 10 15:04 tmp

Thanks in advance,
Peter
-- 
Peter Lindgren, dataingenj?r
E-post: peter "at" norrskenkonsult punkt com
http://www.norrskenkonsult.com

Update: I did a kernel trace to see what the imap process was doing, and
it was retrieving headers, but to me it seemed like there were an awful
lot of null (\0) and control characters among the headers. Excerpts of
the trace below, actual mail address hidden to protect the innocent.

It's difficult to trace a folder that works, since that imap process
doesn't live long enough for me to trace it. Hints on that, other than
being really quick on the keyboard? ;-) The traces I've managed has nice
plaintext commands like these:
   2994 imap     1165779379.972538 GIO   fd 1 wrote 23 bytes
        "24 OK NOOP completed.\r
        "
   2994 imap     1165779379.972620 RET   write 23/0x17
...
   2994 imap     1165779379.977092 CALL  read(0,0x6be31b,0xce5)
   2994 imap     1165779379.977179 GIO   fd 0 read 29 bytes
        "25 UID fetch 3657:* (FLAGS)\r
        "
   2994 imap     1165779379.977245 RET   read 29/0x1d


However, these reads/writes are to other file descriptors (0 and 1) than
the erraneous reads below (fd 8).


My system:
# uname -ap
OpenBSD casiopea.norrskenkonsult.com 3.7 GENERIC#312 sparc
SUNW,SPARCstation-5, MB86907 @ 170 MHz, on-chip FPU



# kdump -T -f imap.tracelog
   9506 imap     1165775763.392950 EMUL  "native"
   9506 imap     1165775763.521181 CALL  munmap(0xbef8000,0x100000)
   9506 imap     1165775763.526077 RET   munmap 0
   9506 imap     1165775763.563349 CALL  munmap(0xbeb7000,0x41000)
   9506 imap     1165775763.564797 RET   munmap 0
   9506 imap     1165775763.564899 CALL  munmap(0xbeb6000,0x1000)
   9506 imap     1165775763.565074 RET   munmap 0
   9506 imap     1165775763.565192 CALL
pread(0x8,0x8580000,0x1000,0,0,0x141000)
   9506 imap     1165775763.566267 GIO   fd 8 read 4088 bytes
        "\0\0\0\^\\0\0\0\0In-Reply-To:
<1137747953.43d0a7f16017b at webbmail.thornqvist.se>
         \0\0\0\0\^N\0\0\0Q\0\0\0\^W\0\0\0\0Message-ID:
<6996e6f30601200332y1f7dcb9fi1397e97e41b9c04 at mail.gmail.com>
         \0\0\0\0\0\0\^Q\0\0\0t\0\0\0 \0\0\0\0References:
<WinLMSG.43cfec50.1783.5ecb at smtp.chello.se.>
                 <1137747953.43d0a7f16017b at webbmail.thornqvist.se>
         \0\0\0\^R\0\0\0*\0\0\0\^[\0\0\0\0Subject: Re: [exakt] Pod-TV i SVT
         \0\0\0\0\0\^S\0\0\0.\0\0\0\^Z\0\0\0\0To: Exakt IT
<exakt at lists.frobbit.se>
         \0\0\0\0\0      \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0
         \M-D\0\0\0\0\0\0

\M-w\0\0\0\0\0\0\^DG\0\0\0\0\0\0\^Dd\0\0\0\^]\0\^A3\M-X\0\0\^B\M-@\0\0\0
         \0\0\0\\\0\0\0\^[\0\0\0\0Content-Type: text/plain;
format=flowed; charset="iso-8859-1";
                 reply-type=response
         \0\0\0\v\0\0\0.\0\0\0\^Y\0\0\0\0Date: Fri, 20 Jan 2006 12:34:59
+0100
         \0\0\0\0\0\f\0\0\0003\0\0\0\^T\0\0\0\0From: "Lars Ilshammar"
<xxxx at xxxxxxxxx.se>
         \0\0\0\0\^N\0\0\0;\0\0\0\^S\0\0\0\0Message-ID:
<004301c61db5$8b847e40$6400a8c0 at LASSE>
         \0\0\0\0\^Q\0\0\0o\0\0\0\^V\0\0\0\0References:
<82ab3b60601192318q7c5d7cf1u6ee17edc0e11da92 at mail.gmail.com>
                 <43D09D62.2080005 at abm.uu.se>
         \0\0\0\0\^R\0\0\0)\0\0\0\^X\0\0\0\0Subject: Re: [exakt] SMHI suger!
         \0\0\0\0\0\0\^S\0\0\0000\0\0\0\^U\0\0\0\0To: "Exakt IT"
<exakt at lists.frobbit.se>
         \0\0\0\^T\0\0\0\^V\0\0\0\^^\0\0\0\0X-Priority: 3
         \0\0\0\0\0      \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0\b\M-<\0\0\0\0\0\0\b\M-l\0\0\0\0\0\0\^O\M-~\0\0\0\0\0\0\^PZ\0\0\0\\\0\
         \^A3\M-l\0\0\^B\M-T\0\0\0
         \0\0\0F\0\0\0\^X\0\0\0\0Content-Type: text/plain;
format=flowed; charset="iso-8859-1"
         \0\0\0\0\0\v\0\0\0.\0\0\0\^V\0\0\0\0Date: Fri, 20 Jan 2006
13:06:34 +0100

... continues like that until ...

   9506 imap     1165775763.566267 GIO   fd 8 read 8 bytes
        "\0\0\0\f\0\0\0H"
   9506 imap     1165775763.566912 RET   pread 4096/0x1000
   9506 imap     1165775763.570811 CALL
mmap(0,0x144000,0x3,0x1002,0xffffffff,0,0,0)
   9506 imap     1165775763.570972 RET   mmap 256024576/0xf42a000
   9506 imap     1165775763.715347 CALL  munmap(0x8481000,0x100000)
   9506 imap     1165775763.719472 RET   munmap 0
   9506 imap     1165775763.757083 CALL  munmap(0x843f000,0x42000)
   9506 imap     1165775763.758449 RET   munmap 0
   9506 imap     1165775763.758569 CALL  munmap(0x843e000,0x1000)
   9506 imap     1165775763.758735 RET   munmap 0
   9506 imap     1165775763.758854 CALL
pread(0x8,0xf56d000,0x1000,0,0,0x142000)
   9506 imap     1165775763.759294 GIO   fd 8 read 4088 bytes

...

/Peter
-- 
Peter Lindgren, dataingenj?r
E-post: peter "at" norrskenkonsult punkt com
http://www.norrskenkonsult.com