dovecot - Aug 2015 - Duplicate mails with pop3 + dsync replication

It turns out that if I enable this option:
pop3_deleted_flag = "$POP3Deleted?

The issue no longer persists. I have to manually expunge the kept mails that
have been deleted via pop3 though:
doveadm expunge mailbox INBOX KEYWORD '$POP3Deleted' -A

Wolfgang
> On 21 Jun 2015, at 21:05, Wolfgang Hennerbichler <wogri at wogri.com>
wrote:
> 
> FWIW I just tried the sdbox mailbox format for a singe user (not the whole
installation) by overriding the mail-userattr via ldap and the problem persists,
so I don?t think it?s an issue with maildir per se, but rather a pop3 /
replication issue or a misconfiguration on my side.
> 
> Wolfgang
> 
>> On 20 Jun 2015, at 22:29, Wolfgang Hennerbichler <wogri at
wogri.com> wrote:
>> 
>> Hi,
>> 
>> When I enable (and it only happens when it is actually running)
replication over TCP my users get duplicate mails via pop3.
>> Here?s what happens: 
>> 
>> * User gets mail
>> * Mail appears in (Maildir) new/ folder on both replicas
>> * user retrieves mail (retr), calls dele. Mail still exists in new/
folder on both replicas.
>> * user runs retr (and gets the duplicate) and dele a second time, the
mail is actually removed.
>> If I disable replication this behaviour goes away. 
>> I use dovecot from debian stable resources. 
>> Any help would _really_ be appreciated. I found one german thread that
dealt with the same problem, but no solution for it
(http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-replikation/)
>> 
>> wogri
>> 
>> # dovecot --version
>> 2.2.13
>> 
>> # dovecot -n 
>> # 2.2.13: /etc/dovecot/dovecot.conf
>> # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4
>> auth_verbose = yes
>> doveadm_password = XXX
>> first_valid_gid = 113
>> first_valid_uid = 108
>> last_valid_gid = 113
>> last_valid_uid = 108
>> mail_gid = dovecot
>> mail_location = maildir:/var/mail/%n
>> mail_plugins = quota fts fts_lucene virtual notify replication
>> mail_temp_dir = /var/lib/dovecot/tmp
>> mail_uid = dovecot
>> namespace {
>> list = children
>> location = virtual:/var/mail/%n/virtual
>> prefix = virtual.
>> separator = .
>> }
>> namespace inbox {
>> inbox = yes
>> list = yes
>> location = 
>> mailbox "Deleted Messages" {
>>   auto = subscribe
>>   special_use = \Trash
>> }
>> mailbox Drafts {
>>   special_use = \Drafts
>> }
>> mailbox Junk {
>>   special_use = \Junk
>> }
>> mailbox Sent {
>>   special_use = \Sent
>> }
>> mailbox "Sent Messages" {
>>   auto = no
>>   special_use = \Sent
>> }
>> mailbox Spam {
>>   auto = subscribe
>>   special_use = \Junk
>> }                                                                      
[48/4646]
>> mailbox Trash {
>>   special_use = \Trash
>> }
>> prefix = 
>> separator = .
>> subscriptions = yes
>> type = private
>> }
>> passdb {
>> args = /etc/dovecot/dovecot-ldap.conf.ext
>> driver = ldap
>> }
>> plugin {
>> default_language = de
>> fts = lucene
>> fts_lucene = whitespace_chars=@.
>> mail_replica = tcp:172.16.1.1:12345
>> quota = maildir:User quota
>> quota_rule = *:storage=4G
>> quota_rule2 = Trash:storage=+100M
>> quota_rule3 = Spam:ignore
>> quota_warning = storage=95%% quota-warning 95 %u
>> quota_warning2 = storage=80%% quota-warning 80 %u
>> sieve = /etc/sieve/%n.sieve
>> sieve_default = /etc/sieve/default.sieve
>> sieve_dir = ~/sieve
>> sieve_extensions = +editheader
>> }
>> postmaster_address = postmaster at wogri.at
>> protocols = " imap lmtp pop3"
>> service aggregator {
>> fifo_listener replication-notify-fifo {
>>   user = dovecot
>> }
>> unix_listener replication-notify {
>>   user = dovecot
>> }
>> }
>> service doveadm {
>> inet_listener {
>>   port = 12345
>> }
>> }
>> service imap {
>> process_limit = 1024
>> }
>> }
>> service lmtp {
>> inet_listener lmtp {
>>   address = 127.0.0.1
>>   port = 2003
>> }
>> user = dovecot
>> }
>> service managesieve-login {
>> inet_listener sieve {
>>   port = 4190
>> }
>> service_count = 1
>> }
>> service pop3 {
>> process_limit = 1024
>> }
>> service quota-warning {
>> executable = script /usr/local/sbin/quota-warning.sh
>> unix_listener quota-warning {
>>   user = dovecot
>> }
>> user = dovecot
>> }
>> service replicator {
>> process_min_avail = 1
>> unix_listener replicator-doveadm {
>>   mode = 0600
>> }
>> }
>> ssl = required
>> ssl_cert = </etc/exim4/server.cert.pem
>> ssl_key = </etc/exim4/server.key.pem
>> userdb {
>> args = /etc/dovecot/dovecot-ldap.conf.ext
>> driver = ldap
>> }
>> protocol lmtp {
>> mail_plugins = quota sieve virtual notify replication
>> }
>> protocol lda {
>> mail_plugins = quota fts fts_lucene virtual notify replication sieve
>> }
>> protocol imap {
>> mail_max_userip_connections = 50
>> mail_plugins = quota fts fts_lucene virtual notify replication
imap_quota
>> }
>> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20150622/92e1a58e/attachment.p7s>

On 22/6/2015 9:49 am, Wolfgang Hennerbichler wrote:
> It turns out that if I enable this option:
> pop3_deleted_flag = "$POP3Deleted?
>
> The issue no longer persists. I have to manually expunge the kept mails
that have been deleted via pop3 though:
> doveadm expunge mailbox INBOX KEYWORD '$POP3Deleted' -A
Hello,

I was wondering if the same behavior persists with latest version(s), 
for example 2.2.18.

If so, how do you overcome this issue? What is the suggested practice?

Can someone please provide administration guidelines to handle it?

I am seeing this behavior on v2.2.13; I am now planning an upgrade to 
2.2.18.

Please advise!

My setup is:

Master 1:

    dsync_remote_cmd = ssh -l root master2.example.com doveadm
    dsync-server -u%u
    replication_dsync_parameters = -d -N -l 30 -U

    service aggregator {
       fifo_listener replication-notify-fifo {
         user = vmail
       }
       unix_listener replication-notify {
         user = vmail
       }
    }

    service replicator {
        unix_listener replicator-doveadm {
          mode = 0600
        }
    }

Master 2:

    dsync_remote_cmd = ssh -l root master1.example.com doveadm
    dsync-server -u%u
    replication_dsync_parameters = -d -N -l 30 -U

    service aggregator {
       fifo_listener replication-notify-fifo {
         user = vmail
       }
       unix_listener replication-notify {
         user = vmail
       }
    }

    service replicator {
       unix_listener replicator-doveadm {
         mode = 0600
       }
    }

Replication plugin is enabled on protocols pop3, imap, lda.

Thanks in advance,
Nick

On 17/8/2015 2:29 ??, Wolfgang Hennerbichler wrote:
>> I was wondering if the same behavior persists with latest version(s),
for example 2.2.18.
> I don?t know, I have to deal with the packages that come with my
distribution?s repository.
>
>> >If so, how do you overcome this issue? What is the suggested
practice?
> All I can say is that it works for me with the above setting and a cron-job
that runs every minute which expunges mails for the pop3 users.
>
Thank you for your reply. I am on CentOS 5 and 6, but I am building RPMs 
for our test and production servers.

I recently built v2.2.18 and I decided to give it a try. (I run it with 
postfix - also just upgraded to 2.11.6.)

The issue seems to not occur any more.

I had already tested on two test servers, and I hoped someone could 
provide independent feedback before I decided to upgrade production 
servers. However, I now also upgraded our two production servers and 
things seem to be running smoothly.

All the best,
Nick