dovecot - Jul 2010 - performance problems with {s,m}dbox

using rc3 and with the latest pigeonhole, I had problems with dovecot
(deliver) keeping up with high volumes of E-mail. I tried sdbox, mdbox, and
cyrdir all with similar (failed) result. using the
default Maildir I had no problems. I did not try mbox. The problems seemed
to happen when there were multiple deliver's running to the same folder I
had defined in my sieve rules, either it would timeout and fail, or timeout
and get delivered to my inbox, crash,  or eventually halt because the queue
would reach 20,000 messages which is the maximum with my postfix
configuration.

Everything was fine when I was using Maildir, no errors and no sieve
errors/timeouts. perhaps it is more of an issue with pigeonhole than sieve,
but thought I would post my experience using dbox, since it sounded like it
was a winner based on performance.


Jul 29 06:59:12 db1 dovecot: lda(XXXX): Error: Timeout while waiting for
lock for transaction log file /home/XXXX/Maildir.mdbox/storage/dovecot.map
.index.log
Jul 29 06:59:12 db1 dovecot: lda(XXXX): Error: sieve: msgid=<
20100729135337.D7BBA3A03B3 at db1.XXXXXXX.COM>: failed to store into mailbox
'XXXX': Internal error occurred. Refer to server log for more
information.
[2010-07-29 06:56:12]
Jul 29 06:59:27 db1 dovecot: lda(XXXX): Error: sieve: execution of script
/home/XXXX/.dovecot.sieve failed, but implicit keep was successful

Jul 29 07:13:53 db1 dovecot: lda(XXXX): Error: Timeout while waiting for
lock for transaction log file /home/XXXX/Maildir.mdbox/storage/dovecot.map
.index.log
Jul 29 07:13:53 db1 dovecot: lda(XXXX): Error: sieve: msgid=<
20100729140329.9CBD73A0334 at db1.XXXXXxxxxXXXX.com>: failed to store into
mailbox 'XXXX': Internal error occurred. Refer to server log for more
information. [2010-07-29 07:10:53]
Jul 29 07:14:18 db1 dovecot: lda(XXXX): Error: sieve: execution of script
/home/XXXX/.dovecot.sieve failed, but implicit keep was successful
Jul 29 07:14:18 db1 dovecot: lda(XXXX): Panic: file dbox-file.c: line 76
(dbox_file_free): assertion failed: (file->refcount == 0)
Jul 29 07:14:18 db1 dovecot: lda(XXXX): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x39bba) [0x7f40bf715bba] ->
/usr/lib/dovecot/libdo
vecot.so.0(i_syslog_fatal_handler+0x2b) [0x7f40bf715c4b] ->
/usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f40bf6f31a6] ->
/usr/lib/dovecot/libdov
ecot-storage.so.0(+0x794ac) [0x7f40bfbd74ac] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mdbox_files_free+0x40)
[0x7f40bfbd30b0] -> /usr/lib/doveco
t/libdovecot-storage.so.0(+0x77f89) [0x7f40bfbd5f89] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_unref+0x7d)
[0x7f40bfb87d4d] -> /usr/
lib/dovecot/libdovecot-storage.so.0(+0x23bf9) [0x7f40bfb81bf9] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_namespaces_deinit+0x23)
[0x7f40bfb8
1cd3] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x2bd3d) [0x7f40bfb89d3d]
-> /usr/libexec/dovecot/deliver(main+0x688) [0x402ea8] -> /lib/libc.so
.6(__libc_start_main+0xfd) [0x7f40bef6bc4d] ->
/usr/libexec/dovecot/deliver() [0x4025f9]

On Fri, Jul 30, 2010 at 7:12 PM, Jake Johnson <jakej1978 at gmail.com>
wrote:


> perhaps it is more of an issue with pigeonhole than sieve, but thought I
> would post my experience using dbox, since it sounded like it was a winner
> based on performance.
>
>
> ... i meant to say it may be more of an issue with pigeonhole than dovecot
(not sieve, doh!)

On 31.7.2010, at 3.12, Jake Johnson wrote:
> using rc3 and with the latest pigeonhole, I had problems with dovecot
> (deliver) keeping up with high volumes of E-mail. I tried sdbox, mdbox, and
> cyrdir all with similar (failed) result.
What OS and filesystem?
> Jul 29 06:59:12 db1 dovecot: lda(XXXX): Error: Timeout while waiting for
> lock for transaction log file /home/XXXX/Maildir.mdbox/storage/dovecot.map
> .index.log
This happens only after the file has been locked for 3 minutes. Pretty much the
only reason why that could happen is if there was another process rebuilding the
mdbox storage because it was detected to be corrupted. There should be errors
logged about it too..
> Jul 29 07:14:18 db1 dovecot: lda(XXXX): Panic: file dbox-file.c: line 76
> (dbox_file_free): assertion failed: (file->refcount == 0)
This is a bug in any case.. I'll see if I can figure out why it happened.

Did sdbox log about any errors? Or cydir? Especially cydir is a really really
simple mailbox format and there is no reason why it could be slower than maildir
or do any kind of a long locking.