dovecot - Aug 2012 - dsync and pop3 migration plugin fails for large inbox due to idle timeout

Timo,

There is a problem with the dsync POP3 migration plugin when syncing a large
INBOX. What happens is dsync establishes a connection to the POP3 server at the
beginning of the sync, but then it sits there idle until it's used to sync
the UIDLs. On a very large inbox with many thousands of messages it will be idle
for longer than a POP3 server's idle timeout setting.

When the UIDL sync occurs dsync produces the following errors (because the
connection has been lost due to the POP3 server's inactivity timeout):

dsync(first.last at test.co.nz): Error: pop3c(pop3.test.com): Server
disconnected unexpectedly
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 13: Disconnected
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 14: Disconnected
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 15: Disconnected
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 16: Disconnected
...
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 14492: Disconnected
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 14493: Disconnected
dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get header for
msg 14494: Disconnected

Also at the end of the dsync it actually has an exit status of 0, but it
shouldn't because the POP3 sync failed, and none of the UIDLs are retained.

Dsync should try to keep the POP3 connection alive, by sending a regular
'NOOP' command to the POP3 server, or it should simply reconnect if sees
the connection is lost.

At the moment the only work around is to increase the POP3 server's idle
timeout to a multi hour value, which isn't a good work around as it causes a
lot of inactive POP sessions to bank up.

Regards, Nikolai.

On 3.8.2012, at 10.04, Nikolai Schupbach wrote:
> There is a problem with the dsync POP3 migration plugin when syncing a
large INBOX. What happens is dsync establishes a connection to the POP3 server
at the beginning of the sync, but then it sits there idle until it's used to
sync the UIDLs. On a very large inbox with many thousands of messages it will be
idle for longer than a POP3 server's idle timeout setting.
What should happen is:

1. pop3c connects to server
2. issues LIST and UIDL commands
3. IMAP server issues fetch 1:* RFC822.SIZE (or I guess it does this with a ton
of separate commands)
4. pop3c attempts to do some TOP commands.
> When the UIDL sync occurs dsync produces the following errors (because the
connection has been lost due to the POP3 server's inactivity timeout):
> 
> dsync(first.last at test.co.nz): Error: pop3c(pop3.test.com): Server
disconnected unexpectedly
> dsync(first.last at test.co.nz): Error: pop3_migration: Failed to get
header for msg 13: Disconnected
It looks like the 3 step takes a long time. Do you have
imapc_features=rfc822.size? Also setting mail_prefetch_count to something
between 10..100 should help a lot. And this should also fix it by moving 3 step
to 1: http://hg.dovecot.org/dovecot-2.1/rev/ab6a4455b27d
> Also at the end of the dsync it actually has an exit status of 0, but it
shouldn't because the POP3 sync failed, and none of the UIDLs are retained.
Yes, looks like dsync simply continues without assigning POP3 UIDL. Hmm. Perhaps
it shouldn't do that on temporary failures..
> Dsync should try to keep the POP3 connection alive, by sending a regular
'NOOP' command to the POP3 server,
This isn't easily possible.
> or it should simply reconnect if sees the connection is lost.
It should probably do that, yes.