Hi,
Hope you are well.
We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2),
using dsync to sync the messages between them.
Previously, we used TCPS to sync the messages, but after some testing,
we concluded that syncing over SSH resulted in a lot less failed syncs,
so we started using SSH.
The change has been a success, but I now notice a issue when a email
client immediately moves a messages to a separate folder while Dovecot
is syncing, where the message is seen twice in the folder (with the
exact same headers/content).
For example, I have set a filter in my email client, Thunderbird, to
immediately move all emails coming from this list to a separate folder.
When opening that folder, I do not see one, but two unread messages,
both identical to each other.
When checking the directories on the server, I see this:
mx1:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S
mx2:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
Note the difference: on mx1, one indicates mx1, and one mx2, while on
the other server, both indicate mx2/
Any idea (other then telling end-users not to use such filters) how we
could prevent these duplicate messages?
This is our config:
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
# Hostname: mx1.example.com
auth_mechanisms = plain login
disable_plaintext_auth = no
dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
imap_capability = +SPECIAL-USE XLIST
listen = *,[::]
lmtp_rcpt_check_quota = yes
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = quota
mail_privileged_group = vmail
namespace inbox {
? inbox = yes
? location ? mailbox Drafts {
??? special_use = \Drafts
? }
? mailbox Junk {
??? special_use = \Junk
? }
? mailbox Sent {
??? special_use = \Sent
? }
? mailbox "Sent Messages" {
??? special_use = \Sent
? }
? mailbox Trash {
??? special_use = \Trash
? }
? prefix ? separator = .
}
passdb {
? args = /etc/dovecot/dovecot-sql.conf
? driver = sql
}
plugin {
? mail_replica = remote:root at mx2.example.com
? quota = dict:user::file:/var/vmail/%d/%n/.quotausage
? quota_status_nouser = DUNNO
? quota_status_overquota = 552 5.2.2 Mailbox is full
? quota_status_success = DUNNO
? sieve = /var/vmail/%d/%n/.sieve
? sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
? sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
? sieve_max_actions = 100
? sieve_max_redirects = 25
? sieve_max_script_size = 2M
}
protocols = imap pop3 lmtp
replication_max_conns = 50
service aggregator {
? fifo_listener replication-notify-fifo {
??? mode = 0666
??? user = vmail
? }
? unix_listener replication-notify {
??? mode = 0666
??? user = vmail
? }
}
service auth {
? unix_listener /var/spool/postfix/private/auth {
??? group = postfix
??? mode = 0660
??? user = postfix
? }
? unix_listener auth-userdb {
??? group = vmail
??? mode = 0600
??? user = vmail
? }
? user = root
}
service imap-login {
? client_limit = 1000
? process_limit = 512
}
service lmtp {
? unix_listener /var/spool/postfix/private/dovecot-lmtp {
??? group = postfix
??? mode = 0600
??? user = postfix
? }
}
service quota-status {
? client_limit = 1
? executable = quota-status -p postfix
? unix_listener /var/spool/postfix/private/quota-status {
??? group = postfix
??? mode = 0660
??? user = postfix
? }
}
service replicator {
? process_min_avail = 1
? unix_listener replicator-doveadm {
??? mode = 0666
??? user = vmail
? }
}
service stats {
? unix_listener stats-reader {
??? group = vmail
??? mode = 0660
??? user = vmail
? }
? unix_listener stats-writer {
??? group = vmail
??? mode = 0660
??? user = vmail
? }
}
ssl_cert = </etc/postfix/smtpd.cert
ssl_cipher_list =
kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes
userdb {
? driver = prefetch
}
userdb {
? args = /etc/dovecot/dovecot-sql.conf
? driver = sql
}
protocol imap {
? auth_verbose = yes
? mail_plugins = quota quota imap_quota quota notify replication
}
protocol pop3 {
? auth_verbose = yes
? mail_plugins = quota quota quota notify replication
? pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
? mail_plugins = quota sieve quota sieve quota notify replication
? postmaster_address = postmaster at mx1.example.com
}
protocol lmtp {
? mail_plugins = quota quota sieve sieve quota notify replication
? postmaster_address = postmaster at mx1.example.com
}
Thanks in advance for your thoughts.
Kind regards,
Thom Pol
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20220515/2cd51704/attachment.htm>
Paul Kudla (SCOM.CA Internet Services Inc.)
2022-May-16 12:12 UTC
Duplicate messages if message is moved when using dsync
ok duplicsate emails (even across dsync, replication etc) is typically
handled via a global sieve script
I use :
# cat duplicates.sieve
require "duplicate"; # for dovecot >= 2.2.18
if duplicate {
discard;
stop;
}
for the scripts
and setup sieve to work via my dovecot.conf file
relative parts below :
--------------------------------------------------------------------
protocols = imap pop3 lmtp sieve
protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address = monitor at scom.ca
}
protocol lda {
mail_plugins = $mail_plugins sieve
}
plugin {
.....
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1d
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
.....
}
service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
port = 4190
}
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}
------------------------------------------------------------------
note the sieve_before which handles duplictes during delivery etc.
Happy Monday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266
On 5/15/2022 12:38 PM, Thom Pol wrote:> Hi,
>
> Hope you are well.
>
> We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2),
> using dsync to sync the messages between them.
>
> Previously, we used TCPS to sync the messages, but after some testing,
> we concluded that syncing over SSH resulted in a lot less failed syncs,
> so we started using SSH.
>
> The change has been a success, but I now notice a issue when a email
> client immediately moves a messages to a separate folder while Dovecot
> is syncing, where the message is seen twice in the folder (with the
> exact same headers/content).
>
> For example, I have set a filter in my email client, Thunderbird, to
> immediately move all emails coming from this list to a separate folder.
> When opening that folder, I do not see one, but two unread messages,
> both identical to each other.
>
> When checking the directories on the server, I see this:
> mx1:
>
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
>
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S
>
> mx2:
>
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
>
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
>
> Note the difference: on mx1, one indicates mx1, and one mx2, while on
> the other server, both indicate mx2/
>
> Any idea (other then telling end-users not to use such filters) how we
> could prevent these duplicate messages?
>
> This is our config:
> # Pigeonhole version 0.5.13 (cdd19fe3)
> # OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
> # Hostname: mx1.example.com
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
> imap_capability = +SPECIAL-USE XLIST
> listen = *,[::]
> lmtp_rcpt_check_quota = yes
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_max_userip_connections = 100
> mail_plugins = quota
> mail_privileged_group = vmail
> namespace inbox {
> ? inbox = yes
> ? location > ? mailbox Drafts {
> ??? special_use = \Drafts
> ? }
> ? mailbox Junk {
> ??? special_use = \Junk
> ? }
> ? mailbox Sent {
> ??? special_use = \Sent
> ? }
> ? mailbox "Sent Messages" {
> ??? special_use = \Sent
> ? }
> ? mailbox Trash {
> ??? special_use = \Trash
> ? }
> ? prefix > ? separator = .
> }
> passdb {
> ? args = /etc/dovecot/dovecot-sql.conf
> ? driver = sql
> }
> plugin {
> ? mail_replica = remote:root at mx2.example.com
> ? quota = dict:user::file:/var/vmail/%d/%n/.quotausage
> ? quota_status_nouser = DUNNO
> ? quota_status_overquota = 552 5.2.2 Mailbox is full
> ? quota_status_success = DUNNO
> ? sieve = /var/vmail/%d/%n/.sieve
> ? sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
> ? sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
> ? sieve_max_actions = 100
> ? sieve_max_redirects = 25
> ? sieve_max_script_size = 2M
> }
> protocols = imap pop3 lmtp
> replication_max_conns = 50
> service aggregator {
> ? fifo_listener replication-notify-fifo {
> ??? mode = 0666
> ??? user = vmail
> ? }
> ? unix_listener replication-notify {
> ??? mode = 0666
> ??? user = vmail
> ? }
> }
> service auth {
> ? unix_listener /var/spool/postfix/private/auth {
> ??? group = postfix
> ??? mode = 0660
> ??? user = postfix
> ? }
> ? unix_listener auth-userdb {
> ??? group = vmail
> ??? mode = 0600
> ??? user = vmail
> ? }
> ? user = root
> }
> service imap-login {
> ? client_limit = 1000
> ? process_limit = 512
> }
> service lmtp {
> ? unix_listener /var/spool/postfix/private/dovecot-lmtp {
> ??? group = postfix
> ??? mode = 0600
> ??? user = postfix
> ? }
> }
> service quota-status {
> ? client_limit = 1
> ? executable = quota-status -p postfix
> ? unix_listener /var/spool/postfix/private/quota-status {
> ??? group = postfix
> ??? mode = 0660
> ??? user = postfix
> ? }
> }
> service replicator {
> ? process_min_avail = 1
> ? unix_listener replicator-doveadm {
> ??? mode = 0666
> ??? user = vmail
> ? }
> }
> service stats {
> ? unix_listener stats-reader {
> ??? group = vmail
> ??? mode = 0660
> ??? user = vmail
> ? }
> ? unix_listener stats-writer {
> ??? group = vmail
> ??? mode = 0660
> ??? user = vmail
> ? }
> }
> ssl_cert = </etc/postfix/smtpd.cert
> ssl_cipher_list =
>
kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> ssl_min_protocol = TLSv1.2
> ssl_prefer_server_ciphers = yes
> userdb {
> ? driver = prefetch
> }
> userdb {
> ? args = /etc/dovecot/dovecot-sql.conf
> ? driver = sql
> }
> protocol imap {
> ? auth_verbose = yes
> ? mail_plugins = quota quota imap_quota quota notify replication
> }
> protocol pop3 {
> ? auth_verbose = yes
> ? mail_plugins = quota quota quota notify replication
> ? pop3_uidl_format = %08Xu%08Xv
> }
> protocol lda {
> ? mail_plugins = quota sieve quota sieve quota notify replication
> ? postmaster_address = postmaster at mx1.example.com
> }
> protocol lmtp {
> ? mail_plugins = quota quota sieve sieve quota notify replication
> ? postmaster_address = postmaster at mx1.example.com
> }
>
> Thanks in advance for your thoughts.
>
> Kind regards,
>
> Thom Pol
>
>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>,
and is
> believed to be clean.