dovecot - Sep 2011 - Mails repopping

HI

I don't know if this is a dovecot issue or a client one.  But as Dovecot is
the most recent change, I'll start here.

I have a server that's been running Courier for about 6 years and in all
that time I think I've only ever had 1 issues where an entire mail box was
repopped by a webmail client.  However, since moving to a new server and dovecot
4 weeks ago, I've now had the webmail client repop this account 4 times
(there are about 230 mails in the account).

Is there a setting I need to tighten to prevent/remedy this?  I have no idea if
it's happening on other accounts, but this is one that I see.  The format is
maildir.  There has been no changes to the webmail client.

Thanks.

Simon

> -----Original Message-----
> From: Charles Marcus [mailto:CMarcus at Media-Brokers.com]
> Sent: Friday, September 09, 2011 2:51 PM
> On 2011-09-09 1:07 PM, Simon Brereton <simon.brereton at
buongiorno.com>
> wrote:
> > I have a server that's been running Courier for about 6 years and
> in
> > all that time I think I've only ever had 1 issues where an entire
> mail
> > box was repopped by a webmail client.
> 
> I don't understand...
> 
> Webmail doesn't speak 'pop' (that I've ever heard of)... so
how does
> a webmail client 'repop' emails? What webmail is this?
It's Horde webmail.  Webmail does pop.  Yahoo and Gmail do it too.

When I first set up the server, it did it on first login - obviously because the
server had changed but it's doing it once a week now (and in fact, it did it
twice today).  It's not fatal, no one will die.  It is a PITA though.

Simon

On Fri, 2011-09-09 at 13:07 -0400, Simon Brereton wrote:
> I have a server that's been running Courier for about 6 years and in
> all that time I think I've only ever had 1 issues where an entire mail
> box was repopped by a webmail client.  However, since moving to a new
> server and dovecot 4 weeks ago, I've now had the webmail client repop
> this account 4 times (there are about 230 mails in the account).  
> 
> Is there a setting I need to tighten to prevent/remedy this?  I have
> no idea if it's happening on other accounts, but this is one that I
> see.  The format is maildir.  There has been no changes to the webmail
> client.
dovecot -n output would have been nice. Also do you see anything in
error logs?

If you're using the default pop3_uidl_format it'll rely on IMAP UIDs to
stay the same, and I guess it's possible that due to some other problem
they change (that should be logged as an error/warning though).

You could try setting pop3_uidl_format=%f, but it will cause everyone to
redownload mails. With newer Dovecot versions you could set
pop3_save_uidl=yes and when you think everyone's downloaded mails once
you can safely change the pop3_uidl_format.

Quoting Simon Brereton <simon.brereton at buongiorno.com>:
>> -----Original Message-----
>> From: dovecot-bounces at dovecot.org [mailto:dovecot-
>> bounces at dovecot.org] On Behalf Of Simon Brereton
>> > -----Original Message-----
>> > From: Timo Sirainen [mailto:tss at iki.fi] On Fri, 2011-09-09 at
13:07
>> > -0400, Simon Brereton wrote:
>> >
>> > > I have a server that's been running Courier for about 6
years and
>> > in
>> > > all that time I think I've only ever had 1 issues where
an entire
>> > mail
>> > > box was repopped by a webmail client.  However, since moving
to a
>> > new
>> > > server and dovecot 4 weeks ago, I've now had the webmail
client
>> > repop
>> > > this account 4 times (there are about 230 mails in the
account).
>> > >
>> > > Is there a setting I need to tighten to prevent/remedy this? 
I
>> > have
>> > > no idea if it's happening on other accounts, but this is
one that
>> I
>> > > see.  The format is maildir.  There has been no changes to
the
>> > webmail
>> > > client.
>> >
>> > dovecot -n output would have been nice. Also do you see anything
in
>> > error logs?
>>
>> Ah.  My apologies of course.  Here it is..
>>
>> mail:~# dovecot -n
>> # 1.2.15: /etc/dovecot/dovecot.conf
>> # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 ext3
>> log_timestamp: %Y-%m-%d %H:%M:%S
>> protocols: imap imaps pop3 pop3s
>> ssl_ca_file: /etc/ssl/keys/rhodes-ca.crt
>> ssl_cert_file: /etc/ssl/keys/mail.domain.net.crt
>> ssl_key_file: /etc/ssl/private/mail.domain.net.key
>> disable_plaintext_auth: no
>> login_dir: /var/run/dovecot/login
>> login_executable(default): /usr/lib/dovecot/imap-login
>> login_executable(imap): /usr/lib/dovecot/imap-login
>> login_executable(pop3): /usr/lib/dovecot/pop3-login
>> mail_privileged_group: mailsystem
>> mail_location: maildir:/var/spool/mail/virtual/%d/%n
>> maildir_very_dirty_syncs: yes
>> mbox_write_locks: fcntl dotlock
>> mail_executable(default): /usr/lib/dovecot/imap
>> mail_executable(imap): /usr/lib/dovecot/imap
>> mail_executable(pop3): /usr/lib/dovecot/pop3
>> mail_plugins(default): quota imap_quota
>> mail_plugins(imap): quota imap_quota
>> mail_plugins(pop3): quota
>> mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
>> mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
>> mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
>> imap_client_workarounds(default): outlook-idle delay-newmail
>> imap_client_workarounds(imap): outlook-idle delay-newmail
>> imap_client_workarounds(pop3):
>> pop3_client_workarounds(default):
>> pop3_client_workarounds(imap):
>> pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
>> lda:
>>   postmaster_address: postmaster at domain.net
>>   mail_plugins: quota
>>   log_path:
>>   info_log_path:
>>   deliver_log_format: msgid=%m: %f: %$
>> auth default:
>>   mechanisms: plain login
>>   user: mailsystem
>>   verbose: yes
>>   passdb:
>>     driver: sql
>>     args: /etc/dovecot/dovecot-sql.conf
>>   userdb:
>>     driver: prefetch
>>   userdb:
>>     driver: static
>>     args: uid=999 gid=115 home=/var/spool/mail/virtual/%d/%n
>> allow_all_users=yes
>>   socket:
>>     type: listen
>>     client:
>>       path: /var/spool/postfix/private/auth
>>       mode: 432
>>       user: postfix
>>       group: mailsystem
>>     master:
>>       path: /var/run/dovecot/auth-master
>>       mode: 432
>>       user: mailsystem
>>       group: mailsystem
>> plugin:
>>   quota: maildir
>>
>> Could you make dovecot -n munge the certificate and postmaster email
>> addresses?  I'm not comfortable with that floating on the
internet..
>>
>> The only thing I have in the logs is 2 sessions where mail was popped
>> (note, it doesn't even add up to the 183 messages in the mail box).
>> But those sessions are vastly longer than the regular ones (tens of
>> minutes compared to a few seconds).  Since both IPs are on the back-
>> bone, that's quite a while to download 100 mails (none of which are
>> over
>>
>> Sep 11 21:36:25 mail dovecot: pop3-login: Login:
>> user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 21:36:34 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
>> del=0/183, size=14025971 Sep 11 21:43:44 mail dovecot: pop3-login:
>> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 21:44:54 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
>> del=0/183, size=14025971 Sep 11 21:52:31 mail dovecot: pop3-login:
>> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 22:56:01 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=100/9182678, del=0/183, size=14025971 Sep 11 23:08:58 mail
>> dovecot: pop3-login: Login: user=<user at domain.com>,
method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 23:37:57 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=75/4748674, del=0/183, size=14025971 Sep 12 00:04:11 mail
>> dovecot: pop3-login: Login: user=<user at domain.com>,
method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:04:26 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=0/0, del=0/183, size=14025971 Sep 12 00:07:40 mail dovecot:
>> pop3-login: Login: user=<user at domain.com>, method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:07:53 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=0/0, del=0/183, size=14025971
>>
>>
>> > If you're using the default pop3_uidl_format it'll rely on
IMAP
>> UIDs
>> > to stay the same, and I guess it's possible that due to some
other
>> > problem they change (that should be logged as an error/warning
>> > though).
>> >
>> > You could try setting pop3_uidl_format=%f, but it will cause
>> everyone
>> > to redownload mails. With newer Dovecot versions you could set
>> > pop3_save_uidl=yes and when you think everyone's downloaded
mails
>> once
>> > you can safely change the pop3_uidl_format.
>>
>> Sorry, I'm very new to dovecot and I'm not sure I understand. 
I
>> presume because neither of those keys are in the dovecot -n output
>> that they are as the defaults, yes?  The account is indeed accessed
>> by IMAP as well (from a mobile device mostly), but I don't see
>> anything fishy there either.  How could I see if the IMAP UIDs have
>> changed?
>>
>> Sep 11 21:20:32 mail dovecot: IMAP(user at domain.com): Connection
>> closed bytes=1095/8292
>>
>> Sep 11 21:26:03 mail dovecot: imap-login: Login:
>> user=<user at domain.com>, method=PLAIN, rip=174.252.83.244,
>> lip=83.170.65.xxx, TLS Sep 11 22:11:20 mail dovecot:
>> IMAP(user at domain.com): Disconnected for inactivity bytes=725/5638
Sep
>> 11 22:17:10 mail dovecot: imap-login: Login: user=<user at
domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11
>> 23:12:06 mail dovecot: IMAP(user at domain.com): Disconnected for
>> inactivity bytes=1471/11025 Sep 11 23:23:22 mail dovecot: imap-login:
>> Login: user=<user at domain.com>, method=PLAIN,
rip=174.252.83.244,
>> lip=83.170.65.xxx, TLS Sep 11 23:52:52 mail dovecot:
>> IMAP(user at domain.com): Connection closed bytes=1841/13679 Sep 12
>> 00:08:47 mail dovecot: imap-login: Login: user=<user at
domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
>> 01:19:05 mail dovecot: imap-login: Login: user=<user at
domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
>> 02:57:01 mail dovecot: IMAP(user at domain.com): Connection closed
>> bytes=2713/60026 Sep 12 02:57:01 mail dovecot: IMAP(user at
domain.com):
>> Connection closed bytes=2688/18635
>>
>>
>> There are no errors or warnings in the mail log (I have one shared
>> log file for postfix, amavis and dovecot).  Reading the notes for
>> pop3_save_uidl it doesn't seem to be a dangerous option  - should I
>> turn that on?  Why will it force everyone to redownload mails
>> (there's nothing about it on the wiki)?
>>
>> Thanks!
>>
>> Simon
>
> Any help would be appreciated.
What do you mean by "repopped"?  You mean downloading the entire data
of the messages from the POP3 server?  This is expected behavior when  
using a stateless (e.g. webmail) client.  Kind of the whole reason you  
don't use POP3 in the first place.

Although caching can help.  e.g., Here's what the first connection to  
the server looks like (this is using IMP 5 on a mailstore with 82  
messages):

S (1315951197.4976): +OK Dovecot ready.
C (1315951197.513): [AUTH PLAIN Command - username: slusarz]
S (1315951197.5319): +OK Logged in.
C (1315951197.5325): STAT
S (1315951197.5328): +OK 82 482351
C (1315951197.5348): UIDL
S (1315951197.5354): +OK
S (1315951197.5354): 1 000000014935d409
S (1315951197.5354): 2 000000024935d409
S (1315951197.5354): 3 000000114935d409
[...]
S (1315951197.5363): 82 000000824935d409
S (1315951197.5363): .
C (1315951197.9582): TOP 1 0
S (1315951198.0411): From user at domain.com  Thu Jun 22 11:16:26 2006
[...]
S (1315951198.0416): .
[...]
C (1315951199.0607): LIST
S (1315951199.061): +OK 82 messages:
S (1315951199.061): 1 118630
[...]
S (1315951199.0619): .

We need to grab all headers so we can build the envelope information  
(needed to produce the mailbox listing).  And the LIST command grabs  
the size information (also used in the mailbox listing).

But remember that the full headers will need to be redownloaded  
*EVERY* time you reload the page unless some sort of caching is  
enabled in the client.  That's just the nature of POP3.  (IMAP has the  
same sort of issues - if the stateless client does not cache, the  
envelope information must be downloaded on every access.  However,  
with IMAP, the network traffic is reduced - you can download only the  
needed information, not all header text - and IMAP servers have the  
ability to cache this information behind the scenes due to the  
abstraction of the API.).

This is where caching is pretty much essential on the webmail side.   
If caching is enabled, the best-case scenario is that the the webmail  
server only needs to grab the list of UIDLs on every POP3 server  
access going forward - if the UIDL list has not changed, we know the  
mailbox hasn't changed and the cached information is still valid.  
(CONDSTORE/QRESYNC extensions for IMAP make this synchronization check  
even more efficient in IMAP)

michael