dovecot - Aug 2009 - More effective mailbox fetching over high RTT link

Could you offer some suggestion how to fetch mailbox content over 
high RTT link (with negligible packet loss)?

Currently I use IMAP+IDLE *but* it fails to use full available bandwidth
due to high RTT and "send command wait for response" nature of POP3
and
IMAP4 protocols.

-- 
[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu
Learning without thought is labor lost;
thought without learning is perilous.
  -- Confucius

On Sun, 2009-08-09 at 22:20 +0200, Andrzej Adam Filip
wrote:
> Could you offer some suggestion how to fetch mailbox content over 
> high RTT link (with negligible packet loss)?
> 
> Currently I use IMAP+IDLE *but* it fails to use full available bandwidth
> due to high RTT and "send command wait for response" nature of
POP3 and
> IMAP4 protocols.
I'm not entirely sure what you want. Download all new messages
automatically whenever they show up? And by "mailbox" do you mean
"user"
or "folder"? So would you want to download all new mails in all
folders?
And is it ok to create your own software to do the downloading?

If all of them were "yes", the best you could right now would be to
set
up a virtual "all" mailbox containing all mailboxes. Then in IDLE
you'd
see whenever new messages pop up and then issue FETCH n:* BODY.PEEK[] or
whatever. There's also IMAP NOTIFY extension that would allow your
client to ask server to immediately send the message body instead of the
client having to ask for it. But Dovecot doesn't currently support that.

Or if you just always wanted to download all mails, again a virtual
mailbox and FETCH 1:* BODY.PEEK[] gives you all mails.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090809/773f50b4/attachment-0002.bin>

Timo Sirainen <tss at iki.fi> wrote:
> On Sun, 2009-08-09 at 22:20 +0200, Andrzej Adam Filip wrote:
>> Could you offer some suggestion how to fetch mailbox content over 
>> high RTT link (with negligible packet loss)?
>> 
>> Currently I use IMAP+IDLE *but* it fails to use full available
bandwidth
>> due to high RTT and "send command wait for response" nature
of POP3 and
>> IMAP4 protocols.
>
> I'm not entirely sure what you want. Download all new messages
> automatically whenever they show up?
It is "maximum version" of what I would like to get.
( maximum: IMAP+IDLE equivalent, acceptable: faster POP3)
> And by "mailbox" do you mean "user" or
"folder"?
> So would you want to download all new mails in all folders?
One folder is acceptable. "All folders" would be nice.
> And is it ok to create your own software to do the downloading?
I was thinking about creating perl script for the task.
> If all of them were "yes", the best you could right now would be
to set
> up a virtual "all" mailbox containing all mailboxes. Then in IDLE
you'd
> see whenever new messages pop up and then issue FETCH n:* BODY.PEEK[] or
> whatever. There's also IMAP NOTIFY extension that would allow your
> client to ask server to immediately send the message body instead of the
> client having to ask for it. But Dovecot doesn't currently support
that.
>
> Or if you just always wanted to download all mails, again a virtual
> mailbox and FETCH 1:* BODY.PEEK[] gives you all mails.
-- 
[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu
Men never make passes at girls wearing glasses.
  -- Dorothy Parker

On Sun, 09 Aug 2009 22:20:41 +0200
Andrzej Adam Filip <andrzej.filip at gmail.com> wrote:
> Could you offer some suggestion how to fetch mailbox content over 
> high RTT link (with negligible packet loss)?
> 
> Currently I use IMAP+IDLE *but* it fails to use full available
> bandwidth due to high RTT and "send command wait for response"
nature
> of POP3 and IMAP4 protocols.
The entire purpose of the command tag in IMAP is to facilitate batching
or concurrent commands.  See RFC3501 section 5.5.  It shouldn't be too
difficult to batch requests for new messages, or even select all the
new messages in one request for a single folder.

-- 
Ben Winslow <rain at bluecherry.net>

Andrzej Adam Filip wrote:
> Could you offer some suggestion how to fetch mailbox content over 
> high RTT link (with negligible packet loss)?
>
> Currently I use IMAP+IDLE *but* it fails to use full available bandwidth
> due to high RTT and "send command wait for response" nature of
POP3 and
> IMAP4 protocols.
>
>   
Is this a satellite phone question?

I build a bunch of products to help with this, but for a fairly 
specialist market.  You can probably contact me offlist via this email 
address or any of them on the website below?

    http://www.mailasail.com

However, we use lots of compression, pipelining and some predictive 
fetches to wring every bit of performance out of a 2.4Kbit connection 
with 1.5s+ latency

Good luck

Ed W