dovecot - Jan 2010 - IDLE timing issue - dovecot or fetchmail issue?

Hi guys,

Thanks for the great product.  We've used dovecot for ages as our 
internal mail server and it works great!

I've recently started using fetchmail 6.3.9 (with IDLE enabled) to 
download mail from our ISP (Pair networks).   The ISP is running Dovecot 
1.1.16.  I am unable to get the info about the dovecot config at this time.

The IDLE support in fetchmail allows us to deliver mail near 
instantaneously without the need to poll (which is what we did 
previously using POP3).  However I've noticed an issue where if 2 mails 
are delivered in quick succession fetchmail will only download the first 
email, and the second will not download.   If I then send in another 
test email this triggers the download of 2 emails (the original that 
wasn't downloaded and the test email).

Now I'm wondering whether the issue lies with Dovecot (e.g. a timing 
issue), or is it fetchmail that isn't implementing the IDLE/download 
properly.

I've attached the output of a fetchmail session showing this problem in 
action (I've added some annotations to this file starting with
>>>>>).
The sequence of events are:

    * Start fetchmail with IDLE enabled
    * Send 2 emails in quick succession.
    * Dovecot notifies of the new message
    * Fetchmail downloads the message
    * Fetchmail initiates a new IDLE
    * Then nothing more is delivered. :(
    * I then send another email, which triggers the 2 to download.


I don't know enough about IMAP to understand whether the problem lies 
with Dovecot or Fetchmail.   Hopefully an IMAP expert will be able to 
see from the attached IMAP session.

Regards,

-- 
Matt Doran
PaperCut Software International Pty. Ltd.
http://www.papercut.com/

Phone:   +61 3 9809 5194

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fetchmail.txt
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100112/a26c202a/attachment-0002.txt>

On 12.1.2010, at 4.40, Matt Doran wrote:
> fetchmail: IMAP> A0006 IDLE
> fetchmail: IMAP< + idling
> fetchmail: IMAP< * 1 EXISTS
> fetchmail: IMAP> DONE
> fetchmail: IMAP< * 1 RECENT
> fetchmail: IMAP< A0006 OK Idle completed.
First mail is received, fetchmail starts downloading it.
> fetchmail: IMAP> A0011 STORE 1 +FLAGS (\Seen \Deleted)
> fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen \Recent))
> fetchmail: IMAP< * 2 EXISTS
> fetchmail: IMAP< * 2 RECENT
> fetchmail: IMAP< A0011 OK Store completed.
Second mail is received, Dovecot notifies fetchmail about it.
> fetchmail: IMAP> A0013 IDLE
> fetchmail: IMAP< + idling
Fetchmail is buggy and ignores EXISTS events if they arrive during non-IDLE
commands.

* Matt Doran [2010-01-12 13:40]:
>Hi guys,
>
>Thanks for the great product.  We've used dovecot for ages as our 
>internal mail server and it works great!
Let me second that.
>I've recently started using fetchmail 6.3.9 (with IDLE enabled) to 
>download mail from our ISP (Pair networks).   The ISP is running 
>Dovecot 1.1.16.  I am unable to get the info about the dovecot config 
>at this time.
>
>The IDLE support in fetchmail allows us to deliver mail near 
>instantaneously without the need to poll (which is what we did 
>previously using POP3).
Oh, this is cool.
>However I've noticed an issue where if 2 
>mails are delivered in quick succession fetchmail will only download 
>the first email, and the second will not download.   If I then send 
>in another test email this triggers the download of 2 emails (the 
>original that wasn't downloaded and the test email).
>
>Now I'm wondering whether the issue lies with Dovecot (e.g. a timing 
>issue), or is it fetchmail that isn't implementing the IDLE/download 
>properly.
This is a timing issue, yes, but the IMAP server does what it's supposed
to do - when it sees the first message arrive, it notifies fetchmail
about it. Fetchmail immediately ends IDLE session by sending DONE and
starts fetching that one message. If DONE wasn't sent, the server would
issue another response notifying you about the second message, but it
doesn't get a chance to do so. Please try this: login to server and
enter these commands (to get encrypted connection without using STARTLS
you can use command "openssl s_client -connect mail.XXXX.com:993"):

. login <username> <password>
. select inbox
. idle

Now try send some messages and observe what happens.

When you're done, send these commands to the server

done
. logout

Here's what Dovecot sent to me when idling, I sent 5 messages very
quickly:

* 1 EXISTS
* 2 EXISTS
* 3 EXISTS
* 5 EXISTS

If I'd terminate the IDLE session after the first line, I won't receive
the other lines. So, I'd have to check for new messages myself or start
IDLE session again and wait for yet another message to arrive. Then
Dovecot would tell me about all new messages and I could go on and fetch
them.
>I've attached the output of a fetchmail session showing this problem 
>in action (I've added some annotations to this file starting with 
>>>>>>).  The sequence of events are:
>
>   * Start fetchmail with IDLE enabled
>   * Send 2 emails in quick succession.
>   * Dovecot notifies of the new message
>   * Fetchmail downloads the message
>   * Fetchmail initiates a new IDLE
>   * Then nothing more is delivered. :(
>   * I then send another email, which triggers the 2 to download.
>
>
>I don't know enough about IMAP to understand whether the problem lies 
>with Dovecot or Fetchmail.   Hopefully an IMAP expert will be able to 
>see from the attached IMAP session.
Not being an expert, I tried to provide an explanaition of my
understanding om IDLE. So, in your case it's fetchmail where logic could
be improved.

-- 
     -- Kirill Miazine <km at krot.org>

On Tue, 12 Jan 2010 16:21:17 +0200 Timo Sirainen <tss at iki.fi>
articulated:
> On 12.1.2010, at 4.40, Matt Doran wrote:
> 
> > fetchmail: IMAP> A0006 IDLE
> > fetchmail: IMAP< + idling
> > fetchmail: IMAP< * 1 EXISTS
> > fetchmail: IMAP> DONE
> > fetchmail: IMAP< * 1 RECENT
> > fetchmail: IMAP< A0006 OK Idle completed.
> 
> First mail is received, fetchmail starts downloading it.
> 
> > fetchmail: IMAP> A0011 STORE 1 +FLAGS (\Seen \Deleted)
> > fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen \Recent))
> > fetchmail: IMAP< * 2 EXISTS
> > fetchmail: IMAP< * 2 RECENT
> > fetchmail: IMAP< A0011 OK Store completed.
> 
> Second mail is received, Dovecot notifies fetchmail about it.
> 
> > fetchmail: IMAP> A0013 IDLE
> > fetchmail: IMAP< + idling
> 
> Fetchmail is buggy and ignores EXISTS events if they arrive during non-IDLE
commands.
Out of curiosity, has this been reported to the Fetchmail developer(s)?

-- 
Jerry
gesbbb at yahoo.com