dovecot - Jun 2009 - IMP reading entire mailbox

We have been noticing a constant hum of traffic between our IMAP server
(Dovecot v1.1.14) and our filers over the past several days. I have
narrowed it down to one particular user who has been logged in to
IMP/Horde since the problem began (he hadn't logged since we upgraded to
Dovecot a week before).

IMP is periodically checking this user's mail every five minutes. Each
time it does, the his entire mailbox is being read. I initially thought
it might be an indexing problem since I am also seeing messages such as
this in the logs:

Corrupted index cache file
/local/data/dovecot/indexes/mail/X/XXXXXXXX/.imap/delivery.log/dovecot.index.cache:
Broken MIME parts for mail UID 30880

But the in/out bytes being logged by Dovecot indicate that the entire
contents of his mailbox are being sent to the client. That is, this user
has a particularly large mailbox (~500M) and about 500M are being sent
to IMP every five minutes. (Actually, what is the meaning of the above
message? He's the only user logging it.)

The user does have some filters defined in IMP, but I can't imagine that
it would try to apply them to the entire mailbox each time it is
accessed. Surely it would only apply the filters to new messages, right?
But I can't think of any other reason why we would be seeing this
behaviour, and only for this user.

I have not yet contacted the user as I need something to tell him other
than "you're a bandwidth hog". I think the next step is to ask him
to
disable his filters, but I would prefer not shooting in the dark.

This user alone is adding about 50Mbps of bandwidth; between the time we
upgraded to Dovecot (from uw-imap) and he logged in, we were witnessing
a huge drop in traffic. He's ruining our graphs! ;-)

Seriously, if anyone has some ideas, please let me know.


Thanks!

-- 
Chris O'Regan <chris at encs.concordia.ca>
Senior Unix Systems Administrator, Academic IT Services
Faculty of Engineering and Computer Science
Concordia University, Montreal, Canada

On Jun 9, 2009, at 9:18 AM, Chris O'Regan wrote:
> IMP is periodically checking this user's mail every five minutes. Each
> time it does, the his entire mailbox is being read. I initially  
> thought
> it might be an indexing problem since I am also seeing messages such  
> as
> this in the logs:
>
> Corrupted index cache file /local/data/dovecot/indexes/mail/X/ 
> XXXXXXXX/.imap/delivery.log/dovecot.index.cache: Broken MIME parts  
> for mail UID 30880
Is it always this same UID? How often does it happen? Every 5 minutes?  
Can you reproduce this manually somehow? If there's a way to reproduce  
this, I'd like to get a copy of the mailbox. You can put the mailbox  
through http://dovecot.org/tools/mbox-anonymize.pl and hopefully it  
will then still break.

Dovecot anyway detects that data in cache file is corrupted, so it  
deletes the entire cache file. If IMP actually fetches something like  
BODY or BODYSTRUCTURE for all messages, that causes all the messages  
to be read since the cache was just deleted.

> Corrupted index cache file
/local/data/dovecot/indexes/mail/X/XXXXXXXX/.imap/delivery.log/dovecot.index.cache:
Broken MIME parts for mail UID 30880
DUH! Please ignore the Corrupted index cache file; the "mailbox" in
question is a file named delivery.log (i.e. procmail log file) that is
in the user's ~/mail directory. Of course there will be problems trying
to index this file, it is not an mbox! *sigh*

Okay, I think this rules out an indexing problem. But the main problem
persists, why has IMP all of a sudden begun reading the entire contents
of this user's mailbox each time it connects, and why has the started
since upgrading to Dovecot?

I have another hunch. Time to speak to the user...


Many thanks for the help so far!

-- 
Chris O'Regan <chris at encs.concordia.ca>
Senior Unix Systems Administrator, Academic IT Services
Faculty of Engineering and Computer Science
Concordia University, Montreal, Canada