dovecot - Mar 2012 - Many messages clustered around the same date.saved value

Subject: Different user messages clustered around the same date.saved value

After updating dovecot to 2.1.3, I can now use "doveadm expunge -A
..."
to iterate through all user trash folders and expunge old messages.

However, I noticed a strange thing: querying what would have been deleted

 	doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d

showed many date.saved values are clustered around the same
timestamp, even among different user's Trash mailbox.  One user's trash
mailbox having the same date.saved is explained by a user deleting a
lot of message at one time, but I can't explain why many different users
would have messages with the same (or closeby) date.saved value.

For example, the output of the above query on my system showed the 10s
window /2012-03-05 18:08:0[0-9]/ matched 7658 messages among 22 different
user Trash mailboxes, which is statistically unlikely.

I did't see anything special in the dovecot logs at this time to
explain this.  What would cause this?

Joseph Tam <jtam.home at gmail.com>

On Sun, 2012-03-25 at 00:46 -0700, Joseph Tam wrote:
> Subject: Different user messages clustered around the same date.saved value
> 
> After updating dovecot to 2.1.3, I can now use "doveadm expunge -A
..."
> to iterate through all user trash folders and expunge old messages.
> 
> However, I noticed a strange thing: querying what would have been deleted
> 
>  	doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore
7d
> 
> showed many date.saved values are clustered around the same
> timestamp, even among different user's Trash mailbox.  One user's
trash
> mailbox having the same date.saved is explained by a user deleting a
> lot of message at one time, but I can't explain why many different
users
> would have messages with the same (or closeby) date.saved value.
Which mailbox format? With Maildir the date.saved is taken from
dovecot.index.cache file, and in some cases that might get dropped. If
it does, then it fallbacks to using the file's ctime.

Timo Sirainen wrote:
>> However, I noticed a strange thing: querying what would have been
>> deleted
>> 
>> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore
7d
>> 
>> showed many date.saved values are clustered around the same timestamp,
>> even among different user's Trash mailbox.
>> ...
>> I can't explain why many different users would have messages with
the
>> same (or closeby) date.saved value.
> 
> Which mailbox format? With Maildir the date.saved is taken from
> dovecot.index.cache file, and in some cases that might get dropped.  If
> it does, then it fallbacks to using the file's ctime.
mbox.

A further look into this reveals that the clustered date.saved values
are the earliest values for every mailbox in the system.  This timestamp
is close to the time I was testing "doveadm ... -A", so the likely
explanation is that I accidentally deleted/updated these values using
some variation of doveadm, even though I remember confining my testing
to query/search/fetch.  This appears to be a case of PEBKAC.

These "wrong" values shouldn't cause problems with expunge queries
since
they err on the side of safety.

Thanks for the insight though.

Joseph Tam <jtam.home at gmail.com>

Timo Sirainen <tss at iki.fi> wrote:
>>> Which mailbox format? With Maildir the date.saved is taken from
>>> dovecot.index.cache file, and in some cases that might get dropped.
If
>>> it does, then it fallbacks to using the file's ctime.
>>
>> mbox.
>
> Ah, with mbox there isn't any usable fallback for date.saved.  If
it's
> not in dovecot.index.cache, the current time is used.
I'm a little confused as to why it needed a fallback.  In other words,
why wasn't date.saved put into the index as soon as the IMAP operation
copied it into "Trash"?

If this data isn't set at that time, when does it get instantiated?
When I actually ask for it?

Joseph Tam <jtam.home at gmail.com>