dovecot - Jul 2005 - Dovecot 1.0-stable mbox performance and disconnections

I've got some of our busiest users now using Dovecot instead of UW-IMAP 
with mbox-format mailboxes and predominantly Outlook Express as the client.

Doing some perfomance monitoring (with Solaris 8 process accounting), it 
looks like Dovecot needing to read only 30-50% of the disk blocks needed 
by UW-IMAP, but I was hoping for better!

What seems to happen, is that most of the connections use only a small 
number of blocks (due to Dovecot indexes, presumably) but occassionally 
a process will read lots of blocks, sometimes even more than a UW-IMAP 
session.

Is this expected, or is something going wrong with the index updates? I 
have the default sync settings, I think (mbox_dirty_syncs = yes, 
mbox_very_dirty_syncs = no, mbox_lazy_writes = yes)

My INBOX is an extreme example with ~10,000 messages and most processes 
reading a few hundred blocks but a couple using 20,000 or so!

Also, Dovecot seems to be dropping connections, sometimes, for no good 
reason (putting some extra i_info lines suggests for "inactivity"). 
Thunderbird tends to reconnect automatically, though. It seems it may be 
happening at the end of an IDLE command, something like:-

4 IDLE
+ idling
* 1005 EXISTS
DONE
* 1005 EXPUNGE

but then the expected

4 OK Idle completed

never comes.

Has anybody else seen this?

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094

On Tue, 2005-07-12 at 15:37 +0100, Chris Wakelin wrote:
> Doing some perfomance monitoring (with Solaris 8 process accounting), it 
> looks like Dovecot needing to read only 30-50% of the disk blocks needed 
> by UW-IMAP, but I was hoping for better!
> 
> What seems to happen, is that most of the connections use only a small 
> number of blocks (due to Dovecot indexes, presumably) but occassionally 
> a process will read lots of blocks, sometimes even more than a UW-IMAP 
> session.
Most of this probably happens when new mails are appended to mailbox
using some external tool, and after that the mailbox is selected again
in Dovecot (if new mails arrive to selected mailbox,
mbox_dirty_syncs=yes prevents the mbox from being fully re-read).

Another reason why Dovecot might be reading more than UW-IMAP is that
when setting message flags, it reads the mail headers for those mails,
and later when closing the mailbox it writes the flags. Those extra
reads are completely unnecessary and I should fix the code not to do
that..
> Is this expected, or is something going wrong with the index updates? I 
> have the default sync settings, I think (mbox_dirty_syncs = yes, 
> mbox_very_dirty_syncs = no, mbox_lazy_writes = yes)
Setting mbox_very_dirty_syncs = yes makes Dovecot not re-read the mbox
entirely when selecting externally modified mbox, unless it notices that
mails aren't where they should be.
> Also, Dovecot seems to be dropping connections, sometimes, for no good 
> reason (putting some extra i_info lines suggests for
"inactivity").
> Thunderbird tends to reconnect automatically, though. It seems it may be 
> happening at the end of an IDLE command, something like:-
Hmm.. Dovecot closes the connection because of inactivity? How long has
Thunderbird been idling then? It should be disconnected if it doesn't
say anything for 30 minutes, but not before that.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20050712/6fd7b2cd/attachment-0001.bin>

Timo Sirainen wrote:
> Setting mbox_very_dirty_syncs = yes makes Dovecot not re-read the mbox
> entirely when selecting externally modified mbox, unless it notices that
> mails aren't where they should be.
> 
OK, I've set up another test Dovecot service with
"mbox_very_dirty_syncs
= yes", just for me ;)

I don't think I'd like to risk it with the other users as they stand a 
chance of using UW-IMAP at the same time as Dovecot (e.g. use Webmail 
from home while leaving Outlook Express running on a machine on a 
Dovecot-ed subnet at work).

<snip>
> 
> Hmm.. Dovecot closes the connection because of inactivity? How long has
> Thunderbird been idling then? It should be disconnected if it doesn't
> say anything for 30 minutes, but not before that.
> 
I've just looked at my packet traces again, and it seems it may be 
Thunderbird closing the connection (it's tricky to see as Thunderbird 
will open up to five concurrent connections by default). My colleague 
who runs PC-Pine 4 has had problems, though. I haven't managed to 
reproduce the problem with my "debug" client (telnet). I'll keep
digging...

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094