thr3ads.net - dovecot - Duplicate mails with pop3 + dsync replication [Jun 2015]

If this information is useful, please help other people find it:
Share via:

Wolfgang Hennerbichler

2015-Jun-20 20:29 UTC

Duplicate mails with pop3 + dsync replication

Hi,

When I enable (and it only happens when it is actually running) replication over
TCP my users get duplicate mails via pop3.
Here?s what happens: 

* User gets mail
* Mail appears in (Maildir) new/ folder on both replicas
* user retrieves mail (retr), calls dele. Mail still exists in new/ folder on
both replicas.
* user runs retr (and gets the duplicate) and dele a second time, the mail is
actually removed.
If I disable replication this behaviour goes away. 
I use dovecot from debian stable resources. 
Any help would _really_ be appreciated. I found one german thread that dealt
with the same problem, but no solution for it
(http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-replikation/)

wogri

# dovecot --version
2.2.13

# dovecot -n 
# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4
auth_verbose = yes
doveadm_password = XXX
first_valid_gid = 113
first_valid_uid = 108
last_valid_gid = 113
last_valid_uid = 108
mail_gid = dovecot
mail_location = maildir:/var/mail/%n
mail_plugins = quota fts fts_lucene virtual notify replication
mail_temp_dir = /var/lib/dovecot/tmp
mail_uid = dovecot
namespace {
  list = children
  location = virtual:/var/mail/%n/virtual
  prefix = virtual.
  separator = .
}
namespace inbox {
  inbox = yes
  list = yes
  location = 
  mailbox "Deleted Messages" {
    auto = subscribe
    special_use = \Trash
  }
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    auto = no
    special_use = \Sent
  }
  mailbox Spam {
    auto = subscribe
    special_use = \Junk
  }                                                                             
[48/4646]
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
  separator = .
  subscriptions = yes
  type = private
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  default_language = de
  fts = lucene
  fts_lucene = whitespace_chars=@.
  mail_replica = tcp:172.16.1.1:12345
  quota = maildir:User quota
  quota_rule = *:storage=4G
  quota_rule2 = Trash:storage=+100M
  quota_rule3 = Spam:ignore
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  sieve = /etc/sieve/%n.sieve
  sieve_default = /etc/sieve/default.sieve
  sieve_dir = ~/sieve
  sieve_extensions = +editheader
}
postmaster_address = postmaster at wogri.at
protocols = " imap lmtp pop3"
service aggregator {
  fifo_listener replication-notify-fifo {
    user = dovecot
  }
  unix_listener replication-notify {
    user = dovecot
  }
}
service doveadm {
  inet_listener {
    port = 12345
  }
}
service imap {
  process_limit = 1024
}
}
service lmtp {
  inet_listener lmtp {
    address = 127.0.0.1
    port = 2003
  }
  user = dovecot
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  service_count = 1
}
service pop3 {
  process_limit = 1024
}
service quota-warning {
  executable = script /usr/local/sbin/quota-warning.sh
  unix_listener quota-warning {
    user = dovecot
  }
  user = dovecot
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0600
  }
}
ssl = required
ssl_cert = </etc/exim4/server.cert.pem
ssl_key = </etc/exim4/server.key.pem
userdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
protocol lmtp {
  mail_plugins = quota sieve virtual notify replication
}
protocol lda {
  mail_plugins = quota fts fts_lucene virtual notify replication sieve
}
protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = quota fts fts_lucene virtual notify replication imap_quota
}

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20150620/57c827b1/attachment-0001.p7s>

Wolfgang Hennerbichler

2015-Jun-21 19:05 UTC

head link

Duplicate mails with pop3 + dsync replication

FWIW I just tried the sdbox mailbox format for a singe user (not the whole
installation) by overriding the mail-userattr via ldap and the problem persists,
so I don?t think it?s an issue with maildir per se, but rather a pop3 /
replication issue or a misconfiguration on my side.

Wolfgang
> On 20 Jun 2015, at 22:29, Wolfgang Hennerbichler <wogri at wogri.com>
wrote:
> 
> Hi,
> 
> When I enable (and it only happens when it is actually running) replication
over TCP my users get duplicate mails via pop3.
> Here?s what happens: 
> 
> * User gets mail
> * Mail appears in (Maildir) new/ folder on both replicas
> * user retrieves mail (retr), calls dele. Mail still exists in new/ folder
on both replicas.
> * user runs retr (and gets the duplicate) and dele a second time, the mail
is actually removed.
> If I disable replication this behaviour goes away. 
> I use dovecot from debian stable resources. 
> Any help would _really_ be appreciated. I found one german thread that
dealt with the same problem, but no solution for it
(http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-replikation/)
> 
> wogri
> 
> # dovecot --version
> 2.2.13
> 
> # dovecot -n 
> # 2.2.13: /etc/dovecot/dovecot.conf
> # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4
> auth_verbose = yes
> doveadm_password = XXX
> first_valid_gid = 113
> first_valid_uid = 108
> last_valid_gid = 113
> last_valid_uid = 108
> mail_gid = dovecot
> mail_location = maildir:/var/mail/%n
> mail_plugins = quota fts fts_lucene virtual notify replication
> mail_temp_dir = /var/lib/dovecot/tmp
> mail_uid = dovecot
> namespace {
>  list = children
>  location = virtual:/var/mail/%n/virtual
>  prefix = virtual.
>  separator = .
> }
> namespace inbox {
>  inbox = yes
>  list = yes
>  location = 
>  mailbox "Deleted Messages" {
>    auto = subscribe
>    special_use = \Trash
>  }
>  mailbox Drafts {
>    special_use = \Drafts
>  }
>  mailbox Junk {
>    special_use = \Junk
>  }
>  mailbox Sent {
>    special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>    auto = no
>    special_use = \Sent
>  }
>  mailbox Spam {
>    auto = subscribe
>    special_use = \Junk
>  }                                                                         
[48/4646]
>  mailbox Trash {
>    special_use = \Trash
>  }
>  prefix = 
>  separator = .
>  subscriptions = yes
>  type = private
> }
> passdb {
>  args = /etc/dovecot/dovecot-ldap.conf.ext
>  driver = ldap
> }
> plugin {
>  default_language = de
>  fts = lucene
>  fts_lucene = whitespace_chars=@.
>  mail_replica = tcp:172.16.1.1:12345
>  quota = maildir:User quota
>  quota_rule = *:storage=4G
>  quota_rule2 = Trash:storage=+100M
>  quota_rule3 = Spam:ignore
>  quota_warning = storage=95%% quota-warning 95 %u
>  quota_warning2 = storage=80%% quota-warning 80 %u
>  sieve = /etc/sieve/%n.sieve
>  sieve_default = /etc/sieve/default.sieve
>  sieve_dir = ~/sieve
>  sieve_extensions = +editheader
> }
> postmaster_address = postmaster at wogri.at
> protocols = " imap lmtp pop3"
> service aggregator {
>  fifo_listener replication-notify-fifo {
>    user = dovecot
>  }
>  unix_listener replication-notify {
>    user = dovecot
>  }
> }
> service doveadm {
>  inet_listener {
>    port = 12345
>  }
> }
> service imap {
>  process_limit = 1024
> }
> }
> service lmtp {
>  inet_listener lmtp {
>    address = 127.0.0.1
>    port = 2003
>  }
>  user = dovecot
> }
> service managesieve-login {
>  inet_listener sieve {
>    port = 4190
>  }
>  service_count = 1
> }
> service pop3 {
>  process_limit = 1024
> }
> service quota-warning {
>  executable = script /usr/local/sbin/quota-warning.sh
>  unix_listener quota-warning {
>    user = dovecot
>  }
>  user = dovecot
> }
> service replicator {
>  process_min_avail = 1
>  unix_listener replicator-doveadm {
>    mode = 0600
>  }
> }
> ssl = required
> ssl_cert = </etc/exim4/server.cert.pem
> ssl_key = </etc/exim4/server.key.pem
> userdb {
>  args = /etc/dovecot/dovecot-ldap.conf.ext
>  driver = ldap
> }
> protocol lmtp {
>  mail_plugins = quota sieve virtual notify replication
> }
> protocol lda {
>  mail_plugins = quota fts fts_lucene virtual notify replication sieve
> }
> protocol imap {
>  mail_max_userip_connections = 50
>  mail_plugins = quota fts fts_lucene virtual notify replication imap_quota
> }
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20150621/684ab8df/attachment-0001.p7s>

Wolfgang Hennerbichler

2015-Jun-22 06:49 UTC

head link

Duplicate mails with pop3 + dsync replication

It turns out that if I enable this option:
pop3_deleted_flag = "$POP3Deleted?

The issue no longer persists. I have to manually expunge the kept mails that
have been deleted via pop3 though:
doveadm expunge mailbox INBOX KEYWORD '$POP3Deleted' -A

Wolfgang
> On 21 Jun 2015, at 21:05, Wolfgang Hennerbichler <wogri at wogri.com>
wrote:
> 
> FWIW I just tried the sdbox mailbox format for a singe user (not the whole
installation) by overriding the mail-userattr via ldap and the problem persists,
so I don?t think it?s an issue with maildir per se, but rather a pop3 /
replication issue or a misconfiguration on my side.
> 
> Wolfgang
> 
>> On 20 Jun 2015, at 22:29, Wolfgang Hennerbichler <wogri at
wogri.com> wrote:
>> 
>> Hi,
>> 
>> When I enable (and it only happens when it is actually running)
replication over TCP my users get duplicate mails via pop3.
>> Here?s what happens: 
>> 
>> * User gets mail
>> * Mail appears in (Maildir) new/ folder on both replicas
>> * user retrieves mail (retr), calls dele. Mail still exists in new/
folder on both replicas.
>> * user runs retr (and gets the duplicate) and dele a second time, the
mail is actually removed.
>> If I disable replication this behaviour goes away. 
>> I use dovecot from debian stable resources. 
>> Any help would _really_ be appreciated. I found one german thread that
dealt with the same problem, but no solution for it
(http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-replikation/)
>> 
>> wogri
>> 
>> # dovecot --version
>> 2.2.13
>> 
>> # dovecot -n 
>> # 2.2.13: /etc/dovecot/dovecot.conf
>> # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4
>> auth_verbose = yes
>> doveadm_password = XXX
>> first_valid_gid = 113
>> first_valid_uid = 108
>> last_valid_gid = 113
>> last_valid_uid = 108
>> mail_gid = dovecot
>> mail_location = maildir:/var/mail/%n
>> mail_plugins = quota fts fts_lucene virtual notify replication
>> mail_temp_dir = /var/lib/dovecot/tmp
>> mail_uid = dovecot
>> namespace {
>> list = children
>> location = virtual:/var/mail/%n/virtual
>> prefix = virtual.
>> separator = .
>> }
>> namespace inbox {
>> inbox = yes
>> list = yes
>> location = 
>> mailbox "Deleted Messages" {
>>   auto = subscribe
>>   special_use = \Trash
>> }
>> mailbox Drafts {
>>   special_use = \Drafts
>> }
>> mailbox Junk {
>>   special_use = \Junk
>> }
>> mailbox Sent {
>>   special_use = \Sent
>> }
>> mailbox "Sent Messages" {
>>   auto = no
>>   special_use = \Sent
>> }
>> mailbox Spam {
>>   auto = subscribe
>>   special_use = \Junk
>> }                                                                      
[48/4646]
>> mailbox Trash {
>>   special_use = \Trash
>> }
>> prefix = 
>> separator = .
>> subscriptions = yes
>> type = private
>> }
>> passdb {
>> args = /etc/dovecot/dovecot-ldap.conf.ext
>> driver = ldap
>> }
>> plugin {
>> default_language = de
>> fts = lucene
>> fts_lucene = whitespace_chars=@.
>> mail_replica = tcp:172.16.1.1:12345
>> quota = maildir:User quota
>> quota_rule = *:storage=4G
>> quota_rule2 = Trash:storage=+100M
>> quota_rule3 = Spam:ignore
>> quota_warning = storage=95%% quota-warning 95 %u
>> quota_warning2 = storage=80%% quota-warning 80 %u
>> sieve = /etc/sieve/%n.sieve
>> sieve_default = /etc/sieve/default.sieve
>> sieve_dir = ~/sieve
>> sieve_extensions = +editheader
>> }
>> postmaster_address = postmaster at wogri.at
>> protocols = " imap lmtp pop3"
>> service aggregator {
>> fifo_listener replication-notify-fifo {
>>   user = dovecot
>> }
>> unix_listener replication-notify {
>>   user = dovecot
>> }
>> }
>> service doveadm {
>> inet_listener {
>>   port = 12345
>> }
>> }
>> service imap {
>> process_limit = 1024
>> }
>> }
>> service lmtp {
>> inet_listener lmtp {
>>   address = 127.0.0.1
>>   port = 2003
>> }
>> user = dovecot
>> }
>> service managesieve-login {
>> inet_listener sieve {
>>   port = 4190
>> }
>> service_count = 1
>> }
>> service pop3 {
>> process_limit = 1024
>> }
>> service quota-warning {
>> executable = script /usr/local/sbin/quota-warning.sh
>> unix_listener quota-warning {
>>   user = dovecot
>> }
>> user = dovecot
>> }
>> service replicator {
>> process_min_avail = 1
>> unix_listener replicator-doveadm {
>>   mode = 0600
>> }
>> }
>> ssl = required
>> ssl_cert = </etc/exim4/server.cert.pem
>> ssl_key = </etc/exim4/server.key.pem
>> userdb {
>> args = /etc/dovecot/dovecot-ldap.conf.ext
>> driver = ldap
>> }
>> protocol lmtp {
>> mail_plugins = quota sieve virtual notify replication
>> }
>> protocol lda {
>> mail_plugins = quota fts fts_lucene virtual notify replication sieve
>> }
>> protocol imap {
>> mail_max_userip_connections = 50
>> mail_plugins = quota fts fts_lucene virtual notify replication
imap_quota
>> }
>> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20150622/92e1a58e/attachment.p7s>

Apparently Analagous Threads

Search for more apparently analagous threads

dovecot - Jun 2015 - Duplicate mails with pop3 + dsync replication

Duplicate mails with pop3 + dsync replication

Duplicate mails with pop3 + dsync replication

Duplicate mails with pop3 + dsync replication

Apparently Analagous Threads