dovecot - Feb 2007 - Differences between IDLE when Maildir vs. wel the "mail" folder is used

Are there any differences with how both IDLE and "UID FETCH n:n+x (FLAGS
UID...)" are handled when Dovecot is using a "mail" vs. a
"Maildir"
user?

I have recently implemented support for IDLE (Push E-mail) in a client
infrastructure that I'm building. I haven't in depth checked out what
the problem might be, but when Dovecot was using the "mail" dir in
$HOME
something about the implementation wasn't working. When I switched to
"Maildir" everything worked. 

I only tested to the point of seeing the "* EXISTS n" response during
the idle. I saw this response. It's for example possible that the n is
wrong or that a bit later the "UID FETCH ..." ain't returning the
right
information. It's also possible that my implementation is wrong (of
course) and I haven't yet had the time to really figure out what goes
wrong.

I did so far, however, test with Cyrus, Exchange, Courier and Isode's
M-Box. All these tests where successful. SO if it indeed is a
discrepancy in my implementation .. it's going to be something very
specialised or specific I fear :-)

I also setup testing infrastructure 
http://tinymail.org/trac/tinymail/wiki/TestImapServers


Also take a careful look at this manual test. "tnytest" is a user on
imap2.tinymail.org that uses Maildir. The "tnytest2" user doesn't
have a
Maildir configured. By that I mean that "Dovecot" has created the
folder
for E-mail using whatever it uses by default.

Dovecot's IDLE (the EXISTS) responses also take a lot longer on tnytest2
than on tnytest.

Another question. Why not simply "always" use the Maildir solution if
that one works better? ;-) (really, honest question. Hehe).

Take a look at the "UID FETCH" response: It's incorrect on
tnytest2. The
first attempt I asked for 1:1 and I got none. The second I asked for 1:4
and I got 3 of them, while IDLE clearly stated that there where 4
messages available. In reality 4 where indeed available (or copied in
total).

Well, I'm not saying something is or isn't wrong, just wondering and
testing a little bit and seeing some strange results...

Feel free to use imap2.tinymail.org for testing. And if you need root on
that virtual machine, so that you can install a new version, go ahead.
On http://imap2.tinymail.org you van find things like version
information (it's a CVS snapshot).


pvanhoof at lort:~$ telnet imap2.tinymail.org
Trying 86.39.154.77...
telnet: Unable to connect to remote host: Connection refused
pvanhoof at lort:~$ telnet imap2.tinymail.org 143
Trying 86.39.154.77...
Connected to imap2.tinymail.org.
Escape character is '^]'.
* OK Dovecot ready.
ao1 login tnytest tnytest
ao1 OK Logged in.
a01 select INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1171049606] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
a01 OK [READ-WRITE] Select completed.
a04 IDLE
+ idling
* 2 EXISTS
DONE
a04 OK Idle completed.
UID FETCH 1:2 (FLAGS UID)
* 1 FETCH (FLAGS (\Seen) UID 1)
* 2 FETCH (FLAGS (\Seen) UID 2)
UID OK Fetch completed.
^]

telnet> quit
Connection closed.
pvanhoof at lort:~$ telnet imap2.tinymail.org 143
Trying 86.39.154.77...
Connected to imap2.tinymail.org.
Escape character is '^]'.
* OK Dovecot ready.
a01 login tnytest2 tnytest
a01 OK Logged in.
a01 select INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1171107413] UIDs valid
* OK [UIDNEXT 1] Predicted next UID
a01 OK [READ-WRITE] Select completed.
a04 IDLE
+ idling
* 1 EXISTS
DONE
a04 OK Idle completed.
a05 UID FETCH 1:1 (FLAGS UID)
a05 OK Fetch completed.
a04 IDLE
+ idling
* 4 EXISTS
DONE
a04 OK Idle completed.
a06 UID FETCH 1:4 (FLAGS UID)
* 1 FETCH (FLAGS (\Seen) UID 2)
* 2 FETCH (FLAGS (\Seen) UID 3)
* 3 FETCH (FLAGS (\Seen) UID 4)
a06 OK Fetch completed.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog

On Sat, 2007-02-10 at 12:58 +0100, Philip Van Hoof
wrote:
> Dovecot's IDLE (the EXISTS) responses also take a lot longer on
tnytest2
> than on tnytest.
That is because with maildir Dovecot can use kernel's dnotify support to
know immediately when a new mail arrives. Inotify apparently supports
also waiting for file changes, so I think it should work if you
configure --with-notify=inotify. I'll have to try this myself too..
> * OK [UIDNEXT 1] Predicted next UID
> a01 OK [READ-WRITE] Select completed.
> a04 IDLE
> + idling
> * 1 EXISTS
> DONE
How did you get those messages into the mailbox? Just wondering why it's
not giving RECENT reply.
> a04 OK Idle completed.
> a05 UID FETCH 1:1 (FLAGS UID)
> a05 OK Fetch completed.
Looks like I can reproduce this. At least with empty mailboxes it seems
that Dovecot gives a new UID to a mail that already had it. I'll go and
fix this.

However, Dovecot's behavior is perfectly valid from IMAP protocol's
point of view. You shouldn't be assuming that the new UID is the same as
UIDNEXT. For example another session could have just appended two
messages and expunged the first one. In that case you'll see exactly the
above behavior from all the IMAP servers.

So you really should be using message sequence numbers instead of UIDs
before you're completely certain what the UID is. And even still using
sequence numbers is a more "correct" IMAP way to do it, plus it's
slightly faster :) I think the UIDs are mostly useful only for some
initial local cache syncing when the mailbox is opened.
> a06 UID FETCH 1:4 (FLAGS UID)
> * 1 FETCH (FLAGS (\Seen) UID 2)
> * 2 FETCH (FLAGS (\Seen) UID 3)
> * 3 FETCH (FLAGS (\Seen) UID 4)
> a06 OK Fetch completed.
This is because the fourth message had UID 5 and you exluded it from the
fetch.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070212/939f0896/attachment.bin>