Hi Dovecot Users,
I?ve configured dovecot dsync replication and I see troubles in the logs and get
user complaints which I can?t explain. I found similar threads on this
mailinglist, but I couldn?t find a solution anywhere. Does anybody have dsync
running without problems on a high volume mailserver?
I see the following logs, examples given:
Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning:
/var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line 10395:
1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 (uid 41092 ->
41093) - retrying by re-reading from beginning
with this one I?m not sure - it might be that this is completely OK because due
to replication UIDs clash. Maybe that?s OK, but I couldn?t find a confirmation.
Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir
/var/mail/zoechi: Expunged message reappeared, giving a new UID (old uid=41092,
file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377)
This one is definitely a problem, deleted messages re-appear in the mailbox.
I made sure that the 2 hosts doing replication have different hostnames. I run
2.2.27 (from debian-jessie backports). Config below.
Thanks for hints (or pointers to other example configs where dsync works without
problems)
# dovecot -n
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.8.14 x86_64 Debian 8.7 ext4
auth_verbose = yes
debug_log_path = /var/log/dovecot.debug
doveadm_password = # hidden, use -P to show it
first_valid_gid = 106
first_valid_uid = 104
hostname = localhost
last_valid_gid = 106
last_valid_uid = 104
mail_gid = dovecot
mail_location = maildir:/var/mail/%n
mail_plugins = quota fts fts_lucene virtual notify replication
mail_temp_dir = /var/lib/dovecot/tmp
mail_uid = dovecot
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext editheader
namespace {
list = children
location = virtual:/var/mail/%n/virtual
prefix = virtual.
separator = .
}
namespace inbox {
inbox = yes
list = yes
location =
mailbox "Deleted Messages" {
auto = subscribe
special_use = \Trash
}
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = .
subscriptions = yes
type = private
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
default_language = de
fts = lucene
fts_lucene = whitespace_chars=@.
mail_replica = tcp:172.16.1.1:12345
quota = maildir:User quota
quota_rule = *:storage=5G
quota_rule2 = Trash:storage=+200M
quota_rule3 = Spam:ignore
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
sieve = /etc/sieve/%n.sieve
sieve_default = /etc/sieve/default.sieve
sieve_dir = ~/sieve
sieve_extensions = +editheader
}
pop3_deleted_flag = $POP3Deleted
postmaster_address = postmaster at wogri.at
protocols = " imap lmtp sieve pop3"
service aggregator {
fifo_listener replication-notify-fifo {
user = dovecot
}
unix_listener replication-notify {
user = dovecot
}
}
service doveadm {
inet_listener {
port = 12345
}
}
service imap {
process_limit = 1024
}
service lmtp {
inet_listener lmtp {
port = 2003
}
unix_listener lmtp {
user = dovecot
}
user = dovecot
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
service_count = 1
}
service pop3 {
process_limit = 1024
}
service quota-warning {
executable = script /usr/local/sbin/quota-warning.sh
unix_listener quota-warning {
user = dovecot
}
user = dovecot
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0600
}
}
ssl = required
ssl_cert = </etc/dovecot/server.cert.pem
ssl_key = # hidden, use -P to show it
userdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = quota sieve virtual notify replication
}
protocol lda {
mail_plugins = quota fts fts_lucene virtual notify replication sieve
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = quota fts fts_lucene virtual notify replication imap_quota
}
protocol pop3 {
mail_plugins = quota fts fts_lucene virtual notify replication
}
local 2a00:XXX::/64 {
doveadm_password = # hidden, use -P to show it
}
On 17 Feb 2017, at 21.37, Wolfgang Hennerbichler <wogri at wogri.com> wrote:> > Hi Dovecot Users, > > I?ve configured dovecot dsync replication and I see troubles in the logs and get user complaints which I can?t explain. I found similar threads on this mailinglist, but I couldn?t find a solution anywhere. Does anybody have dsync running without problems on a high volume mailserver? > > I see the following logs, examples given: > > Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: /var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line 10395: 1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 (uid 41092 -> 41093) - retrying by re-reading from beginning > > with this one I?m not sure - it might be that this is completely OK because due to replication UIDs clash. Maybe that?s OK, but I couldn?t find a confirmation. > > Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir /var/mail/zoechi: Expunged message reappeared, giving a new UID (old uid=41092, file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377)There seems to be something weird with using Maildir and replication. Haven't had time to debug it and it's likely not an easy bug to fix, so for now the solution would be to use only sdbox/mdbox with replication.
On 20/02/2017 9:09 PM, Timo Sirainen wrote:> On 17 Feb 2017, at 21.37, Wolfgang Hennerbichler <wogri at wogri.com> > wrote: >> >> Hi Dovecot Users, >> >> I?ve configured dovecot dsync replication and I see troubles in the >> logs and get user complaints which I can?t explain. I found similar >> threads on this mailinglist, but I couldn?t find a solution >> anywhere. Does anybody have dsync running without problems on a >> high volume mailserver? >> >> I see the following logs, examples given: >> >> Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: >> /var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line >> 10395: 1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 >> (uid 41092 -> 41093) - retrying by re-reading from beginning >> >> with this one I?m not sure - it might be that this is completely OK >> because due to replication UIDs clash. Maybe that?s OK, but I >> couldn?t find a confirmation. >> >> Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir >> /var/mail/zoechi: Expunged message reappeared, giving a new UID >> (old uid=41092, >> file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377) > > There seems to be something weird with using Maildir and replication. > Haven't had time to debug it and it's likely not an easy bug to fix, > so for now the solution would be to use only sdbox/mdbox with > replication.I have experienced this quite a few times. It is tricky to reliably reproduce. I am also using replication with Maildir. This appears to be a timing/locking issue of some sort. One way to trigger this seems to be to have a new mail come in and be delivered, and then as soon as it appears in Thunderbird, delete it straight away. Thunderbird is set to move mails to Trash when they are deleted and expunge the Inbox on exit. My replication partner sees the new mail has come in, and then it pushes it back to the original. Meanwhile the one on the original has already been deleted by the client so the mail re-appears back on the original again. The latency to my replication partner is 150ms or so but this problem might become easier to reproduce if the delay is increased. I can't say that this is definitely how to reproduce it all of the time but when I have experience this, that's roughly the sequence of events. Reuben
On 20 Feb 2017, at 11:09, Timo Sirainen <tss at iki.fi> wrote:> > On 17 Feb 2017, at 21.37, Wolfgang Hennerbichler <wogri at wogri.com> wrote: >> >> Hi Dovecot Users, >> >> I?ve configured dovecot dsync replication and I see troubles in the logs and get user complaints which I can?t explain. I found similar threads on this mailinglist, but I couldn?t find a solution anywhere. Does anybody have dsync running without problems on a high volume mailserver? >> >> I see the following logs, examples given: >> >> Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: /var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line 10395: 1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 (uid 41092 -> 41093) - retrying by re-reading from beginning >> >> with this one I?m not sure - it might be that this is completely OK because due to replication UIDs clash. Maybe that?s OK, but I couldn?t find a confirmation. >> >> Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir /var/mail/zoechi: Expunged message reappeared, giving a new UID (old uid=41092, file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377) > > There seems to be something weird with using Maildir and replication. Haven't had time to debug it and it's likely not an easy bug to fix, so for now the solution would be to use only sdbox/mdbox with replication.Thank you, I can confirm that after the migration to sdbox I don?t see those errors anymore. I could not find good documentation for converting maildir to sdbox on the fly, so I wrote a script. If there?s interest I can share this. wogri