thr3ads.net - dovecot - Dsync induces redundant mail after rapid append/expunge repeatedly [Mar 2016]

If this information is useful, please help other people find it:
Share via:

潘信傑

2016-Mar-01 11:20 UTC

Dsync induces redundant mail after rapid append/expunge repeatedly

?I meet the problem about dsync with ?expunge. The problems is expunged
mail would reappear after dsyncing and it is easily to reproduce.

?Environment Settings:
- Two servers(serverA, serverB) both enabled plugin "*replication"*
- And more detail get by "dovecot -n" show below

?Reproduce flow:
?- ?Run two processes pA and pB both would run in while loop
?- pA?(All behavior is on Draft mailbox and all on serverA):
   - Clean "Draft" mailbox of userA
   - Run the following imap command of userA
         - imap save (save new mail to Draft)
         - imap delete (delete old mail on Draft)
         - imap expunge
        * - Check if draft only have one email (it would be failed in about
50 loops)*
?- pB?(Send mail to serverB):
   - Send new mail to userA (which would be save in INBOX)

?It would be easier reproduce by increasing server loading.?

I have found something
?- T
he expunge changes from server A in dsync connection don't send
immediately, these expunge changes appear after 1~2 dsync connection over.
- The replicated
?
mail is generated from serverB and sync back to serverA so expunged mail
reappeared.
?
dovecot -n (ServerA)
# 2.2.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.77 x86_64  ext4
auth_mechanisms = plain login
default_internal_user = root
default_login_user = dovecot
default_vsz_limit = 0
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
dsync_target_servers = tcps:10.13.23.139:24245
lda_mailbox_autocreate = yes
mail_debug = yes
mail_fsync = never
mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir
mail_max_userip_connections = 30
mail_never_cache_fields = *
mail_plugins = " notify replication  mail_log"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave duplicate
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix }
passdb {
  driver = pam
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_lucene = whitespace_chars=@._/\~&,()[]"
  mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
  mail_log_fields = uid box msgid size flags from subject
}
postmaster_address = postmaster at localhost
protocols = imap pop3 lmtp imap pop3 sieve
replication_dsync_parameters = -N -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
    group = users
    mode = 0666
    user = admin
  }
  unix_listener replication-notify {
    group = users
    mode = 0666
    user = admin
  }
}
service doveadm {
  inet_listener {
    address = 10.13.23.136,127.0.0.1
    port = 24245
    ssl = yes
  }
}
service imap-login {
  inet_listener imap {
    address = 10.13.23.136,127.0.0.1
    port = 143
  }
  inet_listener imaps {
    address = 10.13.23.136,127.0.0.1
    port = 993
    ssl = yes
  }
  process_limit = 256
  service_count = 0
}
service indexer-worker {
  process_limit = 8
}
service pop3-login {
  inet_listener pop3 {
    address = 10.13.23.136,127.0.0.1
    port = 110
  }
  inet_listener pop3s {
    address = 10.13.23.136,127.0.0.1
    port = 995
    ssl = yes
  }
  process_limit = 256
  service_count = 0
}
service replicator {
  unix_listener replicator-doveadm {
    mode = 0666
  }
}
userdb {
  driver = passwd
}
protocol lda {
  mail_plugins = " notify replication  mail_log sieve"
}

dovecot -n (ServerB)
$ ./sbin/dovecot -n

# 2.2.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.77 x86_64
auth_mechanisms = plain login
default_internal_user = root
default_login_user = dovecot
default_vsz_limit = 0
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
dsync_target_servers = tcps:10.13.23.136:24245
lda_mailbox_autocreate = yes
mail_debug = yes
mail_fsync = always
mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir
mail_max_userip_connections = 30
mail_never_cache_fields = *
mail_plugins = " notify replication  mail_log"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational reg
ex imap4flags copy include variables body enotify environment mailbox date
ihave duplicate
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix }
passdb {
  driver = pam
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_lucene = whitespace_chars=@._/\~&,()[]"
  mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
  mail_log_fields = uid box msgid size flags from subject
}
postmaster_address = postmaster at localhost
protocols = imap pop3 lmtp imap pop3 sieve
replication_dsync_parameters = -N -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
    group = users
    mode = 0666
    user = admin
  }
  unix_listener replication-notify {
    group = users
    mode = 0666
    user = admin
  }
}
service doveadm {
  inet_listener {
    address = 10.13.23.139,127.0.0.1
    port = 24245
    ssl = yes
  }
}
service imap-login {
  inet_listener imap {
    address = 10.13.23.139,127.0.0.1
    port = 143
  }
  inet_listener imaps {
    address = 10.13.23.139,127.0.0.1
    port = 993
    ssl = yes
  }
  process_limit = 256
  service_count = 0
}
service indexer-worker {
  process_limit = 4
}
service pop3-login {
  inet_listener pop3 {
    address = 10.13.23.139,127.0.0.1
    port = 110
  }
  inet_listener pop3s {
    address = 10.13.23.139,127.0.0.1
    port = 995
    ssl = yes
  }
  process_limit = 256
  service_count = 0
}
service replicator {
  unix_listener replicator-doveadm {
    mode = 0666
  }
}
userdb {
  driver = passwd
}
protocol lda {
  mail_plugins = " notify replication  mail_log sieve"
}

Regards,
Jay Pan

Timo Sirainen

2016-Mar-01 11:24 UTC

head link

Dsync induces redundant mail after rapid append/expunge repeatedly

On 01 Mar 2016, at 13:20, ??? <sfffaaa at gmail.com>
wrote:> 
> ?I meet the problem about dsync with ?expunge. The problems is expunged
mail would reappear after dsyncing and it is easily to reproduce.
I've never been able to reproduce this easily. But I wonder if this fix
would help here:

https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827
<https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827>

潘信傑

2016-Mar-02 01:04 UTC

head link

Dsync induces redundant mail after rapid append/expunge repeatedly

Hi Timo,

Thanks for your reply,
I've patched this commit before, but it cannot solve my problem.

Besides,
I log more information on server B and it shows server B generates the mail
itself.
(The mail which has UID 2128 is the redundant mail and sync back to server
A)

Feb 24 22:57:36 dsync-server(test 28740): Info: highest_seen_uid 2125,
importer->local_uid_next 2124
Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts:
unwanted UIDs: 2124:2125
Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts:
Reassign UIDs: 2124

Feb 24 22:57:37 dsync-local(test 28748): Debug: brain M: Import Drafts: UID
2125 isn't usable, assigning new UID 2128
Feb 24 22:57:37 dsync-local(test 28748): Info: copy from Drafts:
box=Drafts, uid=2128, msgid=<20151009090132.62413F6136D at
test.yoyo.mail>,
size=34744, from="wf skr" <wfskr at fstml.us>, subject=The wf
skr...,
flags=(\Seen)
Feb 24 22:57:37 dsync-local(test 28748): Info: expunge: box=Drafts,
uid=2125, msgid=<20151009090132.62413F6136D at test.yoyo.mail>,
size=34744,
from="wf skr" <wfskr at fstml.us>, subject=The wf skr...,
flags=(\Seen)

Best Regards,
Jay Pan

2016-03-01 19:24 GMT+08:00 Timo Sirainen <timo.sirainen at dovecot.fi>:
> On 01 Mar 2016, at 13:20, ??? <sfffaaa at gmail.com> wrote:
>
>
> ?I meet the problem about dsync with ?expunge. The problems is expunged
> mail would reappear after dsyncing and it is easily to reproduce.
>
>
> I've never been able to reproduce this easily. But I wonder if this fix
> would help here:
>
>
>
https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827
>
>

Maybe Matching Threads

Search for more seemingly similar threads

dovecot - Mar 2016 - Dsync induces redundant mail after rapid append/expunge repeatedly

Dsync induces redundant mail after rapid append/expunge repeatedly

Dsync induces redundant mail after rapid append/expunge repeatedly

Dsync induces redundant mail after rapid append/expunge repeatedly

Maybe Matching Threads