潘信傑
2016-Mar-01 11:20 UTC
Dsync induces redundant mail after rapid append/expunge repeatedly
?I meet the problem about dsync with ?expunge. The problems is expunged
mail would reappear after dsyncing and it is easily to reproduce.
?Environment Settings:
- Two servers(serverA, serverB) both enabled plugin "*replication"*
- And more detail get by "dovecot -n" show below
?Reproduce flow:
?- ?Run two processes pA and pB both would run in while loop
?- pA?(All behavior is on Draft mailbox and all on serverA):
- Clean "Draft" mailbox of userA
- Run the following imap command of userA
- imap save (save new mail to Draft)
- imap delete (delete old mail on Draft)
- imap expunge
* - Check if draft only have one email (it would be failed in about
50 loops)*
?- pB?(Send mail to serverB):
- Send new mail to userA (which would be save in INBOX)
?It would be easier reproduce by increasing server loading.?
I have found something
?- T
he expunge changes from server A in dsync connection don't send
immediately, these expunge changes appear after 1~2 dsync connection over.
- The replicated
?
mail is generated from serverB and sync back to serverA so expunged mail
reappeared.
?
dovecot -n (ServerA)
# 2.2.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.77 x86_64 ext4
auth_mechanisms = plain login
default_internal_user = root
default_login_user = dovecot
default_vsz_limit = 0
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
dsync_target_servers = tcps:10.13.23.139:24245
lda_mailbox_autocreate = yes
mail_debug = yes
mail_fsync = never
mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir
mail_max_userip_connections = 30
mail_never_cache_fields = *
mail_plugins = " notify replication mail_log"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave duplicate
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix }
passdb {
driver = pam
}
plugin {
fts = lucene
fts_autoindex = yes
fts_lucene = whitespace_chars=@._/\~&,()[]"
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
mail_log_fields = uid box msgid size flags from subject
}
postmaster_address = postmaster at localhost
protocols = imap pop3 lmtp imap pop3 sieve
replication_dsync_parameters = -N -l 30 -U
service aggregator {
fifo_listener replication-notify-fifo {
group = users
mode = 0666
user = admin
}
unix_listener replication-notify {
group = users
mode = 0666
user = admin
}
}
service doveadm {
inet_listener {
address = 10.13.23.136,127.0.0.1
port = 24245
ssl = yes
}
}
service imap-login {
inet_listener imap {
address = 10.13.23.136,127.0.0.1
port = 143
}
inet_listener imaps {
address = 10.13.23.136,127.0.0.1
port = 993
ssl = yes
}
process_limit = 256
service_count = 0
}
service indexer-worker {
process_limit = 8
}
service pop3-login {
inet_listener pop3 {
address = 10.13.23.136,127.0.0.1
port = 110
}
inet_listener pop3s {
address = 10.13.23.136,127.0.0.1
port = 995
ssl = yes
}
process_limit = 256
service_count = 0
}
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
userdb {
driver = passwd
}
protocol lda {
mail_plugins = " notify replication mail_log sieve"
}
dovecot -n (ServerB)
$ ./sbin/dovecot -n
# 2.2.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.77 x86_64
auth_mechanisms = plain login
default_internal_user = root
default_login_user = dovecot
default_vsz_limit = 0
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
dsync_target_servers = tcps:10.13.23.136:24245
lda_mailbox_autocreate = yes
mail_debug = yes
mail_fsync = always
mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir
mail_max_userip_connections = 30
mail_never_cache_fields = *
mail_plugins = " notify replication mail_log"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational reg
ex imap4flags copy include variables body enotify environment mailbox date
ihave duplicate
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix }
passdb {
driver = pam
}
plugin {
fts = lucene
fts_autoindex = yes
fts_lucene = whitespace_chars=@._/\~&,()[]"
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
mail_log_fields = uid box msgid size flags from subject
}
postmaster_address = postmaster at localhost
protocols = imap pop3 lmtp imap pop3 sieve
replication_dsync_parameters = -N -l 30 -U
service aggregator {
fifo_listener replication-notify-fifo {
group = users
mode = 0666
user = admin
}
unix_listener replication-notify {
group = users
mode = 0666
user = admin
}
}
service doveadm {
inet_listener {
address = 10.13.23.139,127.0.0.1
port = 24245
ssl = yes
}
}
service imap-login {
inet_listener imap {
address = 10.13.23.139,127.0.0.1
port = 143
}
inet_listener imaps {
address = 10.13.23.139,127.0.0.1
port = 993
ssl = yes
}
process_limit = 256
service_count = 0
}
service indexer-worker {
process_limit = 4
}
service pop3-login {
inet_listener pop3 {
address = 10.13.23.139,127.0.0.1
port = 110
}
inet_listener pop3s {
address = 10.13.23.139,127.0.0.1
port = 995
ssl = yes
}
process_limit = 256
service_count = 0
}
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
userdb {
driver = passwd
}
protocol lda {
mail_plugins = " notify replication mail_log sieve"
}
Regards,
Jay Pan
Timo Sirainen
2016-Mar-01 11:24 UTC
Dsync induces redundant mail after rapid append/expunge repeatedly
On 01 Mar 2016, at 13:20, ??? <sfffaaa at gmail.com> wrote:> > ?I meet the problem about dsync with ?expunge. The problems is expunged mail would reappear after dsyncing and it is easily to reproduce.I've never been able to reproduce this easily. But I wonder if this fix would help here: https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827 <https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827>
潘信傑
2016-Mar-02 01:04 UTC
Dsync induces redundant mail after rapid append/expunge repeatedly
Hi Timo, Thanks for your reply, I've patched this commit before, but it cannot solve my problem. Besides, I log more information on server B and it shows server B generates the mail itself. (The mail which has UID 2128 is the redundant mail and sync back to server A) Feb 24 22:57:36 dsync-server(test 28740): Info: highest_seen_uid 2125, importer->local_uid_next 2124 Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts: unwanted UIDs: 2124:2125 Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts: Reassign UIDs: 2124 Feb 24 22:57:37 dsync-local(test 28748): Debug: brain M: Import Drafts: UID 2125 isn't usable, assigning new UID 2128 Feb 24 22:57:37 dsync-local(test 28748): Info: copy from Drafts: box=Drafts, uid=2128, msgid=<20151009090132.62413F6136D at test.yoyo.mail>, size=34744, from="wf skr" <wfskr at fstml.us>, subject=The wf skr..., flags=(\Seen) Feb 24 22:57:37 dsync-local(test 28748): Info: expunge: box=Drafts, uid=2125, msgid=<20151009090132.62413F6136D at test.yoyo.mail>, size=34744, from="wf skr" <wfskr at fstml.us>, subject=The wf skr..., flags=(\Seen) Best Regards, Jay Pan 2016-03-01 19:24 GMT+08:00 Timo Sirainen <timo.sirainen at dovecot.fi>:> On 01 Mar 2016, at 13:20, ??? <sfffaaa at gmail.com> wrote: > > > ?I meet the problem about dsync with ?expunge. The problems is expunged > mail would reappear after dsyncing and it is easily to reproduce. > > > I've never been able to reproduce this easily. But I wonder if this fix > would help here: > > > https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b827 > >