thr3ads.net - dovecot - Replication not working - GUIDs conflict

If this information is useful, please help other people find it:
Share via:

Patrick Westenberg

2022-Aug-01 09:15 UTC

Replication not working - GUIDs conflict - will be merged later

Very interesting new insights:

When I use imapsync and let it synchronize mails from INBOX to
INBOX/testfolder, the automatic replication works fine.
All mails are synchronized between my two backends.


When I move the mails to the INBOX (doveadm move -u mail at example.com
INBOX mailbox INBOX/testfolder all), these mails are lost on the
replica! They are neither in INBOX, nor in INBOX/testfolder

Regards
Patrick

Paul Kudla (SCOM.CA Internet Services Inc.)

2022-Aug-01 10:47 UTC

head link

Replication not working - GUIDs conflict - will be merged later

OK thanks for the updates

Long story short i went through a bunch of replication issues when i was 
first setting up dovecot.

Most of which were fixed in 2.3.19 and i have not seen any issues since.

in general i had to turn on the debugging mode (mail_debug = yes)

and filter syslog by "replication"

It was discovered through some work that any account over 300 physical 
folders (give or take) would not replicate (physical size of mailbox had 
nothing to do with the issue just folder count) and would fail without 
error (why i asked the version this was an issue in 2.3.18 and before?)

when a replication sync failed it did say in the logs replication 
requested for <email account> but it would fail and not log the error 
why, replication timeouts were however recorded?

I my self ended up patching the c code in the replicator to get more 
detail how far a replication sync would go (ie i added a bunch of 
logging code to track the issues better)

maybe look at folder counts? - It was only effecting 5 of my customers 
but was a pain to find the issue.

that being said i had to make some scripts to show user replication and 
what was outstanding between the servers (i had to run them on both 
servers seperately to acurately get the replication status going in both 
directions.

I now run these scripts in the background every 5 minutes to make sure 
replication on both sides are in sync.

also i found tcpip replication (without ssl) worked the best? tcpip over 
SSL had timing errors, ssl in my case was not required as the two 
servers were hardwired together (ie no security leaks)

if your two servers are at two different sites consider a small vpn setup

i like using gre for this as it is hardwired by static ip addresses on 
both sides and you can set a 10.x.x.x (or whatever) to communicate 
between the servers ??

also on using the doveadm ssh scripts introduced rights issues between 
the two servers file systems (even though they were identical)

Basically i tried everything !

tcpip - non ssl just seem to work the best.

sync.status :

doveadm replicator status
echo ' '
doveadm replicator dsync-status | grep -v 'Not connected'


which outputs :

# sync.status
Queued 'sync' requests        0 

Queued 'high' requests        0 

Queued 'low' requests         0 

Queued 'failed' requests      0 

Queued 'full resync' requests 0 

Waiting 'failed' requests     0 

Total number of known users   269

oh and another major thing was the replication selecting from the 
database properly.

i use postgresql

#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and
destination = '%u'

iterate_query = SELECT "username" as user, domain FROM email_users
WHERE
status = True and alias_flag = False

note my db setup uses status=True for an active user and 
alias_flag=False for an alias email redirect inside postfix (fyi), you 
can ignore these based on how your database is setup.

dovecot is very intelligent if an active box gets activity on one server 
then it wont replicate to the other server if that mailbox is not 
returned to sync users list (but it will setup / activate replication on 
the server that received the email) - this took a bit to figure out as well.


# cat sync.users
doveadm replicator status '*' | grep '   y'

would only sync what was bad in the replication??

try running

# doveadm user '*' | wc
      269     269    5244

on both servers the account count (269) in my case should be the same on 
both servers.


Here is my dovecot.conf config maybe it will help

replication configs (other the the server ip address) must be the same 
on both sides.


# cat dovecot.conf
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 12.1-RELEASE amd64
# Hostname: mail18.scom.ca

auth_debug = no
auth_debug_passwords = no

default_process_limit = 16384

mail_debug = no

#lock_method = dotlock
#mail_max_lock_timeout = 300s

#mbox_read_locks = dotlock
#mbox_write_locks = dotlock

mmap_disable = yes
dotlock_use_excl = no
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no

auth_mechanisms = plain login
auth_verbose = yes
base_dir = /data/dovecot/run/
debug_log_path = syslog
disable_plaintext_auth = no
dsync_features = empty-header-workaround

info_log_path = syslog
login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c


mail_location = maildir:~/

mail_plugins = " virtual notify replication fts fts_lucene "
mail_prefetch_count = 20

protocols = imap pop3 lmtp sieve


protocol lmtp {
   mail_plugins = $mail_plugins sieve
   postmaster_address = monitor at scom.ca
}

service lmtp {
   process_limit=1000
   vsz_limit = 512m
   client_limit=1
    unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
      group = postfix
      mode = 0600
      user = postfix
   }
}

protocol lda {
   mail_plugins = $mail_plugins sieve
}

service lda {
   process_limit=1000
   vsz_limit = 512m
}

service imap {
   process_limit=4096
   vsz_limit = 2g
   client_limit=1
}

service pop3 {
   process_limit=1000
   vsz_limit = 512m
   client_limit=1
}

namespace inbox {
   inbox = yes
   location    mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox Trash {
     auto = subscribe
     special_use = \Trash
   }
   prefix    separator = /
}

passdb {
   args = /usr/local/etc/dovecot/dovecot-pgsql.conf
   driver = sql
}

doveadm_port = 12345
doveadm_password = secretxxxx

service doveadm {
   process_limit = 0
   process_min_avail = 0
   idle_kill = 0
   client_limit = 1
   user = vmail
   inet_listener {
     port = 12345
   }
}

service config {
   unix_listener config {
     user = vmail
     }
}

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
#dsync_remote_cmd = doveadm sync -d -u%u

replication_dsync_parameters = -d -N -l 300 -U

plugin {
   mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename
   mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
   push_notification_driver = dlog

   sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
   #sieve = ~/.dovecot.sieve
   sieve_duplicate_default_period = 1h
   sieve_duplicate_max_period = 1h
   sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
   sieve_global_dir = /usr/local/etc/dovecot/sieve
   sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve


   mail_replica = tcp:10.221.0.19:12345
   #mail_replica = remote:vmail at 10.221.0.19
   #replication_sync_timeout = 2

   fts = lucene
   fts_lucene = whitespace_chars=@.
         fts_autoindex = yes
         fts_languages = en
}

#sieve_extensions = vnd.dovecot.duplicate

#sieve_plugins = vnd.dovecot.duplicate

service anvil {
   process_limit = 1
   client_limit=5000
   vsz_limit = 512m
   unix_listener anvil {
     group = vmail
     mode = 0666
   }
}

service indexer-worker {
         vsz_limit = 2g
}



service auth {
    process_limit = 1
    client_limit=5000
    vsz_limit = 1g

    unix_listener auth-userdb {
       mode = 0660
       user = vmail
       group = vmail
    }
    unix_listener /var/spool/postfix/private/auth {
       mode = 0666
    }

}

service stats {
   process_limit = 1000
   vsz_limit = 1g
   unix_listener stats-reader {
     group = vmail
     mode = 0666
   }
   unix_listener stats-writer {
     group = vmail
     mode = 0666
   }
}
userdb {
   args = /usr/local/etc/dovecot/dovecot-pgsql.conf
   driver = sql

}

protocol imap {
   mail_max_userip_connections = 50
   mail_plugins = $mail_plugins notify replication
}

protocol pop3 {
   mail_max_userip_connections = 50
   mail_plugins = $mail_plugins notify replication
}

protocol imaps {
   mail_max_userip_connections = 25
   mail_plugins = $mail_plugins notify replication
}

protocol pop3s {
   mail_max_userip_connections = 25
   mail_plugins = $mail_plugins notify replication
}


service managesieve-login {
   process_limit = 1000
   vsz_limit = 1g
   inet_listener sieve {
     port = 4190
   }
}

verbose_proctitle = yes

replication_max_conns = 100

replication_full_sync_interval = 1d

service replicator {
   client_limit = 0
   drop_priv_before_exec = no
   idle_kill = 4294967295s
   process_limit = 1
   process_min_avail = 0
   service_count = 0
   vsz_limit = 8g
         unix_listener replicator-doveadm {
     mode = 0600
     user = vmail
   }
   vsz_limit = 8192M
}


service aggregator {
   process_limit = 1000
   #vsz_limit = 1g
   fifo_listener replication-notify-fifo {
     user = vmail
     group = vmail
     mode = 0666
   }

}

service pop3-login {
   process_limit = 1000
   client_limit = 100
   vsz_limit = 512m
  }


service imap-urlauth-login {
   process_limit = 1000
   client_limit = 1000
   vsz_limit = 1g
}


service imap-login {
   process_limit=1000
   client_limit = 1000
   vsz_limit = 1g
  }


protocol sieve {
   managesieve_implementation_string = Dovecot Pigeonhole
   managesieve_max_line_length = 65536
}




#Addition ssl config
!include sni.conf

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname= user= passworddefault_pass_scheme =
PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status =
True and
destination = '%u'

user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and
destination = '%u'

#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and
destination = '%u'

iterate_query = SELECT "username" as user, domain FROM email_users
WHERE
status = True and alias_flag = False

Please note the above selects return a full email address (how i do it) 
- ie paul at scom.ca



# cat sni.conf
#sni.conf
ssl = yes
verbose_ssl = yes
ssl_dh =</usr/local/etc/dovecot/dh-4096.pem
ssl_prefer_server_ciphers = yes
#ssl_min_protocol = TLSv1.2

#Default *.scom.ca
ssl_key =</usr/local/etc/dovecot/scom.pem
ssl_cert =</usr/local/etc/dovecot/scom.pem
ssl_ca =</usr/local/etc/dovecot/scom.pem

local_name .scom.ca {
   ssl_key = /programs/common/getssl.cert -c *.scom.ca -q yes
   ssl_cert = /programs/common/getssl.cert -c *.scom.ca -q yes
   ssl_ca = /programs/common/getssl.cert -c *.scom.ca -q yes
}

local_name mail.clancyca.com {
   ssl_key = /programs/common/getssl.cert -c mail.clancyca.com -q yes
   ssl_cert = /programs/common/getssl.cert -c mail.clancyca.com -q yes
   ssl_ca = /programs/common/getssl.cert -c mail.clancyca.com -q yes
}

local_name mail.paulkudla.net {
   ssl_key = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
   ssl_cert = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
   ssl_ca = /programs/common/getssl.cert -c mail.paulkudla.net -q yes
}

local_name secure.clancyca.com {
   ssl_key = /programs/common/getssl.cert -c secure.clancyca.com -q yes
   ssl_cert = /programs/common/getssl.cert -c secure.clancyca.com -q yes
   ssl_ca = /programs/common/getssl.cert -c secure.clancyca.com -q yes
}

local_name mail.ekst.ca {
   ssl_key = /programs/common/getssl.cert -c mail.ekst.ca -q yes
   ssl_cert = /programs/common/getssl.cert -c mail.ekst.ca -q yes
   ssl_ca = /programs/common/getssl.cert -c mail.ekst.ca -q yes
}

local_name mail.hamletdevelopments.ca {
   ssl_key = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes
   ssl_cert = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes
   ssl_ca = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes
}

note the sni.conf above suck in the certs from a db.

Another thought is are you running duplicate supression, i am not sure 
how that would work when using imapsync (ie i have to assume a lot of 
emails when you run a sync would carry the same info)

duplicate suppression seems to pickup on job numbers, to, from etc to 
decide if an email is a duplicate. Maybe this is also an issue.

# cat duplicates.sieve
require "duplicate";           # for dovecot >= 2.2.18

if duplicate {
     discard;
     stop;
}




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266
Email?paul at scom.ca

On 8/1/2022 5:15 AM, Patrick Westenberg wrote:> 
> Very interesting new insights:
> 
> When I use imapsync and let it synchronize mails from INBOX to
> INBOX/testfolder, the automatic replication works fine.
> All mails are synchronized between my two backends.
> 
> 
> When I move the mails to the INBOX (doveadm move -u mail at example.com
> INBOX mailbox INBOX/testfolder all), these mails are lost on the
> replica! They are neither in INBOX, nor in INBOX/testfolder
> 
> Regards
> Patrick
>

Sebastian Marske

2022-Aug-02 13:30 UTC

head link

Replication not working - GUIDs conflict - will be merged later

On 8/1/22 11:15, Patrick Westenberg wrote:> Very interesting new insights:
> 
> When I use imapsync and let it synchronize mails from INBOX to
> INBOX/testfolder, the automatic replication works fine.
> All mails are synchronized between my two backends.
> 
> 
> When I move the mails to the INBOX (doveadm move -u mail at example.com
> INBOX mailbox INBOX/testfolder all), these mails are lost on the
> replica! They are neither in INBOX, nor in INBOX/testfolder
> 
> Regards
> Patrick
Hi,

every now and then I have the same problem on our servers. Currently,
I'm running Dovecot 2.3.19.1 as well, but I upgraded directly from
2.3.16 due to other issues with the versions in between.

Last time I observed a de-sync due to a GUID change, it appeared like
the user had moved a folder around in their mailbox. And indeed, the
output of 'doveadm mailbox status -u someuser guid '*' listed
different
GUIDs. Dovecot actually logged some errors for this case:

Dovecot log from replica1:
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10206><s1aFMQ8O4WLeJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
b0053e390f0ee162de270000c9042436 to path/to/folder
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10208><fgWCCRAO4WLgJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
5823fe0d100ee162e0270000c9042436 to path/to/folder

Dovecot log from replica2:
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2239><TD9EDAwO4WK/CAAADKIhQg>: Warning: Failed to
do
incremental sync for mailbox path/to/folder, retry with a full sync
(uidnext 1 < 13)
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2241><ix0uKQwO4WLBCAAADKIhQg>: Error: Duplicate
mailbox GUID 0ccaab01079031620e1e00000ca22142 for mailboxes
path/to/folder and some/folder - giving a new GUID
78c9dc2c0c0ee162c10800000ca22142 to path/to/folder

At that time, only replica2 was accepting imap connections.
In this particular case, Dovecot eventually managed to get things back
in sync after way over 24h, but I also had users out of sync for
multiple days.
Running 'doveadm -Dv sync -u someuser -d' manually gave me the same
error message, but didn't change anything.

Other things I've observed:
* it's not limited to a fixed set of users (unlike the
too-many-folders-thing with Dovecot 2.3.1[78])
* it's not limited to newly created users, but also affects users, that
have been in sync for months/years
* it's not limited to mailboxes with lots of imap operations going on
* it's not specific to very large or very small mailboxes (although I've
only seen it for folders with a small number of mails in them)
* in most cases, Dovecot doesn't log any errors
* it does seem to be related to something an imap client can trigger

As of now, my "fix" is to
* make sure that one of the replicas has all mails for that folder
(we're using maildir, so I can just rsync the individual mails/folders)
* create a full copy of the complete folder as backup
* remove the user from replication
* 'doveadm mailbox delete' the folder on one replica to get
rid of one of the conflicting guids (one time, Dovecot replicated the
deletion despite removing the user from replication, so the backup came
in handy)
* alternatively, you might be fine by deleting the folder's index files
* add the user back to replication
* let dsync replicate the user
-> fixed

It's not a very convenient way to resolve this, but maybe it helps. Any
better solutions are greatly appreciated!


Best
Sebastian

dovecot - Aug 2022 - Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later