dovecot - Mar 2006 - phantom new mail (mailboxes keep on being reported as having new mail)

Hi,

I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a
problem of what I call 'phantom mail'.  This problem I have for a long
time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem
still is in there.

My setup is as follows:
- regular x86 box
- running Exim and Dovecot
- Exim saves new mail in maildir format in a local filesystem
  /mail/user/blabla
- Dovecot reads and serves IMAP from local filesystem /mail/user/blabla
- client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows)
  via IMAP

Situation that happens:
I get new mail in a folder, say =news (in Mutt style, it's .news dir in
the maildir directory).  I read the item, Mutt (or Thunderbird) marks
the item as read (moving it over to cur aswell I think), save the
folder, then move on to another folder.  At this point dovecot IMAP
server indicates to the client that there is new mail in the folder
=news.  If I visit the folder with the client, I see no new mail, and
sometimes I have two or three of those folders, and I can keep on
cycling with Mutt till I get bored :).
Before upgrading Dovecot, I knew of no way to resolve this problem;
eventually, it seemed that at some point the folder seemed to figure out
itself that there was indeed no new mail.  Sometimes I could trick it by
letting Mutt mark the mail as new again, and then save the folder.
Didn't always work.
Now, I think a week or two ago I upgraded Dovecot and had to manually
move all the .subscription files, and I also in one go decided to remove
the dovecot* and .imap* files.  That cleaned up, and at first that
looked like it solved my problem.  Not.  So I figured out throwing away
dovecot's index stuff (rm dovecot* in the directory that shows phantom
mail) resolves the problem.  However, this method starts to get annoying
so to say.

So, I did some investigation and found out that the dovecot-uidlist file
seems to be not up-to-date on a folder that shows phantom mails:

# awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids
# ls cur | sort | diff -u - tmp.uids
--- -   2006-03-08 17:45:26.428168000 +0100
+++ tmp.uids    2006-03-08 17:45:19.000000000 +0100
@@ -47,4 +47,5 @@
 1141340181.H111425P8973.hermes.orakel.ods.org:2,S
 1141387968.P13010Q0M35974.hermes:2,S
 1141528391.H769531P12886.hermes.orakel.ods.org:2,S
-1141830301.H349821P23427.hermes.orakel.ods.org:2,S
+1141544380
+1141830301.H349821P23427.hermes.orakel.ods.org

It seems the dovecot uidlist doesn't have the right name for the file on
disk, probably therefore "seeing" a new file all the time and
reporting
the folder as having new mail.  The 1141544380 entry comes from the
first row of the dovecot-uidlist file:
1 1141544380 51
I guess it's just some sort of line to keep track of the last timestamp
and the number of messages or something.  If my theory is right here
then it looks as if the 'bad entry' is actually more recent than this
most recent timestamp reports.  But this is only speculation of course.

Enough for this lenghty mail I guess.  Is this a known issue, or can it
be caused by the way I set up my system?  I must note that this only
happens for some folders, not all of them.  I have folders which get a
lot of mail, but don't have this happening on.  Any clues go resolve
this are welcome.


Regards,

-- 
Fabian Groffen

This problem gets rather annoying.  Isn't there anything that I can try
to resolve this?  It dovecot can't work so well with its indices, is
there an option to disable them?  I keep on deleting those files to get
rid of phantom new mail.


On 09-03-2006 20:21:26 +0100, Grobian wrote:
> Hi,
> 
> I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a
> problem of what I call 'phantom mail'.  This problem I have for a
long
> time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem
> still is in there.
> 
> My setup is as follows:
> - regular x86 box
> - running Exim and Dovecot
> - Exim saves new mail in maildir format in a local filesystem
>   /mail/user/blabla
> - Dovecot reads and serves IMAP from local filesystem /mail/user/blabla
> - client is Mutt and/or Thunderbird from any OS (Mac, Linux, Windows)
>   via IMAP
> 
> Situation that happens:
> I get new mail in a folder, say =news (in Mutt style, it's .news dir in
> the maildir directory).  I read the item, Mutt (or Thunderbird) marks
> the item as read (moving it over to cur aswell I think), save the
> folder, then move on to another folder.  At this point dovecot IMAP
> server indicates to the client that there is new mail in the folder
> =news.  If I visit the folder with the client, I see no new mail, and
> sometimes I have two or three of those folders, and I can keep on
> cycling with Mutt till I get bored :).
> Before upgrading Dovecot, I knew of no way to resolve this problem;
> eventually, it seemed that at some point the folder seemed to figure out
> itself that there was indeed no new mail.  Sometimes I could trick it by
> letting Mutt mark the mail as new again, and then save the folder.
> Didn't always work.
> Now, I think a week or two ago I upgraded Dovecot and had to manually
> move all the .subscription files, and I also in one go decided to remove
> the dovecot* and .imap* files.  That cleaned up, and at first that
> looked like it solved my problem.  Not.  So I figured out throwing away
> dovecot's index stuff (rm dovecot* in the directory that shows phantom
> mail) resolves the problem.  However, this method starts to get annoying
> so to say.
> 
> So, I did some investigation and found out that the dovecot-uidlist file
> seems to be not up-to-date on a folder that shows phantom mails:
> 
> # awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids
> # ls cur | sort | diff -u - tmp.uids
> --- -   2006-03-08 17:45:26.428168000 +0100
> +++ tmp.uids    2006-03-08 17:45:19.000000000 +0100
> @@ -47,4 +47,5 @@
>  1141340181.H111425P8973.hermes.orakel.ods.org:2,S
>  1141387968.P13010Q0M35974.hermes:2,S
>  1141528391.H769531P12886.hermes.orakel.ods.org:2,S
> -1141830301.H349821P23427.hermes.orakel.ods.org:2,S
> +1141544380
> +1141830301.H349821P23427.hermes.orakel.ods.org
> 
> It seems the dovecot uidlist doesn't have the right name for the file
on
> disk, probably therefore "seeing" a new file all the time and
reporting
> the folder as having new mail.  The 1141544380 entry comes from the
> first row of the dovecot-uidlist file:
> 1 1141544380 51
> I guess it's just some sort of line to keep track of the last timestamp
> and the number of messages or something.  If my theory is right here
> then it looks as if the 'bad entry' is actually more recent than
this
> most recent timestamp reports.  But this is only speculation of course.
> 
> Enough for this lenghty mail I guess.  Is this a known issue, or can it
> be caused by the way I set up my system?  I must note that this only
> happens for some folders, not all of them.  I have folders which get a
> lot of mail, but don't have this happening on.  Any clues go resolve
> this are welcome.
> 
> 
> Regards,
> 
-- 
Fabian Groffen
Gentoo for Mac OS X Project

On Thu, 2006-03-09 at 20:21 +0100, Grobian wrote:
> I'm using dovecot 1.0_beta3 on Gentoo Linux, and am experiencing a
> problem of what I call 'phantom mail'.  This problem I have for a
long
> time (pre 1.0), but at the moment I'm using 1.0_beta3 and the problem
> still is in there.
The problem was that STATUS command synchronized the index and maildir,
so that the next time mailbox was opened Dovecot thought the new/
directory was already synchronized so it didn't bother looking into it.
But that also meant that it didn't move the mails to cur/ and their
recent flag state was kept..

Fixed now in CVS, and upcoming beta4.
> # awk '{ print $2 }' dovecot-uidlist | sort > tmp.uids
> # ls cur | sort | diff -u - tmp.uids
> --- -   2006-03-08 17:45:26.428168000 +0100
> +++ tmp.uids    2006-03-08 17:45:19.000000000 +0100
> @@ -47,4 +47,5 @@
>  1141340181.H111425P8973.hermes.orakel.ods.org:2,S
>  1141387968.P13010Q0M35974.hermes:2,S
>  1141528391.H769531P12886.hermes.orakel.ods.org:2,S
> -1141830301.H349821P23427.hermes.orakel.ods.org:2,S
> +1141544380
> +1141830301.H349821P23427.hermes.orakel.ods.org
> 
> It seems the dovecot uidlist doesn't have the right name for the file
on
> disk, 
This is done on purpose. The filenames keep changing and only the base
filename is important. Dovecot just tries to optimize file opening a bit
by keeping the last known full filename in the dovecot-uidlist file.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20060402/4864fcf4/attachment.bin>