thr3ads.net - dovecot - Replication not working - GUIDs conflict

If this information is useful, please help other people find it:
Share via:

Patrick Westenberg

2022-Jul-31 12:16 UTC

Replication not working - GUIDs conflict - will be merged later

Hi everyone,

I have a weird problem with replication that I'm unable to solve.

A new account is sychronized from an external provider via imapsync.
The mails end up on my backend1. I see that the folder structure is
immediately replicated to backend2.

However, a lot of mails are missing and "doveadm replicator status"
also
states that something failed:


                            	priority fast sync full sync success sync
failed
mail at example.com none     00:02:36  00:39:47  -            y

There are no error log entries regarding this user at all.

After some research I tried to start the replication manually with
"doveadm -D backup -u mail at example.com -d tcp:x.y.z.11" and finally
I got an error message:

Jul 31 13:55:37
doveadm(mail at example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>: Debug:
brain
M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1,
remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be
merged later

As a matter of fact, the mails in the inbox are the ones that are
missing on backend2.


I always end up with this error, no matter what I do:
- I completely removed the folder structure on backend2.
- I removed all database entries.
- I removed this user from the replication, ran imapsync again followed
by "doveadm -D backup ..."

I also removed fts-flatcurve to rule this out as a cause of error.

I have no idea what to do. This configuration worked for months.
I got this error with 2.3.17.1 and 2.3.19.1

Any hints would be highly appreciated.


Regards
Patrick





# 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
auth_master_user_separator = *
auth_mechanisms = plain login
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = x.y.z.12
lmtp_save_to_detail_mailbox = yes
log_debug = category=fts-flatcurve
log_path = /var/log/dovecot.log
mail_debug = yes
mail_home = /srv/mail/%Ld/%Ln
mail_location = maildir:~/Maildir
mail_plugins = quota notify replication
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext editheader
metric imapc_traffic {
  fields = bytes_in bytes_out
  filter = event=imap_command_finished
  group_by = user
}
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Spamverdacht {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}
passdb {
  args = /usr/local/etc/dovecot/master-user
  driver = passwd-file
  master = yes
  pass = yes
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  mail_replica = tcp:x.y.z.11
  quota = count:User quota
  quota_grace = 25M
  quota_rule2 = INBOX/Trash:storage=+100M
  quota_status_nouser = DUNNO
  quota_status_success = DUNNO
  quota_vsizes = yes
  quota_warning = storage=100%% quota-full %u
  quota_warning2 = storage=95%% quota-warning 95 %u
  quota_warning3 = storage=80%% quota-warning 80 %u
  quota_warning4 = -storage=100%% quota-ok %u
  sieve = ~/.dovecot.sieve
  sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve
  sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
  sieve_dir = ~/sieve
  sieve_extensions = +editheader
}
protocols = imap pop3 lmtp sieve
service aggregator {
  fifo_listener replication-notify-fifo {
    user = vmail
  }
  unix_listener replication-notify {
    user = vmail
  }
}
service auth {
  unix_listener auth-userdb {
    group = vmail
    mode = 0666
    user = vmail
  }
}
service config {
  unix_listener config {
    mode = 0600
    user = vmail
  }
}
service doveadm {
  inet_listener {
    port = 12345
  }
}
service imap-login {
  process_min_avail = 2
  service_count = 0
}
service lmtp {
  executable = lmtp -L
  inet_listener lmtp {
    address = x.y.z.12
    port = 24
  }
  process_min_avail = 20
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
service quota-full {
  executable = script /usr/local/etc/dovecot/quota_full.sh
  unix_listener quota-full {
    user = vmail
  }
  user = root
}
service quota-ok {
  executable = script /usr/local/etc/dovecot/quota_ok.sh
  unix_listener quota-ok {
    user = vmail
  }
  user = root
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  inet_listener {
    port = 12340
  }
}
service quota-warning {
  executable = script /usr/local/etc/dovecot/quota_warning.sh
  unix_listener quota-warning {
    user = vmail
  }
  user = root
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0600
    user = vmail
  }
}
ssl = required
ssl_cert = </etc/ssl/certs/bundle.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  driver = prefetch
}
userdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
verbose_proctitle = yes
protocol lmtp {
  info_log_path = /var/log/dovecot-lmtp.log
  mail_plugins = quota notify replication sieve
}
protocol imap {
  mail_max_userip_connections = 40
  mail_plugins = quota notify replication imap_quota
}

Paul Kudla (SCOM.CA Internet Services Inc.)

2022-Jul-31 12:47 UTC

head link

Replication not working - GUIDs conflict - will be merged later

Ok This is speculation but i understand the issue at a programming level

what needs to be understood is that imap's uids & ugid's are
relative to
the host server the email is coming from.

this is generally not an issue with replication on cyrus or dovecot 
because the server and the replication is being handled by the same 
server set (ie the same uids & guids etc are generated as things happen)

example replicated data :

-rw-------    1 vmail  vmail  uarch  185K Jul 29 09:30 
1659101404.M875201P20192.mail19.scom.ca,S=189252,W=192431:2,S
-rw-------    1 vmail  vmail  uarch  1.5K Jul 29 09:53 
1659102818.M268117P41331.mail18.scom.ca,S=1583,W=1639:2,S
-rw-------    1 vmail  vmail  uarch  1.0M Jul 29 12:52 
1659113530.M841469P58214.mail18.scom.ca,S=1095861,W=1113817:2,S
-rw-------    1 vmail  vmail  uarch  210K Jul 29 13:15 
1659114913.M958008P31982.mail19.scom.ca,S=215405,W=219216:2,S

you will note the originating server is in the mail file name (mail19 & 
mail18 in my case)

this is how dovecot sorts out the uids etc on the fly. (i think)

If i have read this correctly you are trying to sync to an external imap 
server that carries its own uids guids etc which will be different.

where you are saying that you are using imap sync i assume you are using 
the unix version

# imapsync
Name:

  imapsync - Email IMAP tool for syncing, copying, migrating and archiving
  email mailboxes between two imap servers, one way, and without duplicates.

Version:

  This documentation refers to Imapsync $Revision: 1.977 $


if so look at the

--useuid            :


Use UIDs instead of headers as a criterion to recognize
messages. Option --usecache is then implied unless
                         --nousecache is used.


and the --logfile (ie run a logging file when connecting the the 
external account), it might help with any errors being generated (run 
imapsync in debug mode to get full detail)

basically using useuid deals with sometimes getting a different uid back 
from the origional server

i go through this issue more with pop3 as it returns the id list 
starting at uid 1 (for example) instead of the actual uid against the 
email on the server.

uids will force a proper sync (imap or pop3) because the uid on the 
server will always return the same uid for that email message and 
increments forward inside the account.

if so then imap sync should be sorting this out when syncing the imap 
accounts ? (ie creating new usid guids etc)

so assuming the above is happening the next question is are you using 
replication that is fully setup between the two servers or are you doing 
manual replication (ie running the doveadm command to do the sync?)

(you mentioned using the backup command which would kinda work but full 
replication does the changes on the fly and should work)

if you are running manual replication you should consider going to the 
live replication, it will sort out stuff as the imap folders sync etc. 
(or it should)

The next thing to consider is there were some issues that were fixed in 
2.3.19 replication, are you running the same dovecot versions on both 
servers ?

I do a ton of emails, reporting etc and find that replication works well 
on dovecot 2.3.19 bewteen both of my mail servers. ie it does not matter 
which one receives the email it gets sorted out. If there is an error 
the replication will sort it out on the next sync run through the 
replication process running in the background.

you can set all of the retries etc for replication in the config files.

the merged later is probably indicating that dovecot will sort stuff out 
in the background (ie a reindex etc) but that is putting extra stress on 
the server(s), i used to get the merge or duplicate uids, guids on cyrus 
and it would try to sort it out on the fly. this would occur when one 
replicated server was offline and i was forcing a sync update after 
bringing it back online, this was the case because both servers had 
received emails into the same account from seperate sources thus the 
same uid was set for two different messages on each server. (fyi)

with syrus a rebuild was the only was to sort this out

dovecot seems way more resiliant in this department.


again full replication setup would sort these issue out i expect as each 
server would handle stuff as it happens and adjust uid,guids accordingly.







Happy Sunday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266
Email?paul at scom.ca

On 7/31/2022 8:16 AM, Patrick Westenberg wrote:> 
> Hi everyone,
> 
> I have a weird problem with replication that I'm unable to solve.
> 
> A new account is sychronized from an external provider via imapsync.
> The mails end up on my backend1. I see that the folder structure is
> immediately replicated to backend2.
> 
> However, a lot of mails are missing and "doveadm replicator
status" also
> states that something failed:
> 
> 
>                              	priority fast sync full sync success sync
> failed
> mail at example.com none     00:02:36  00:39:47  -            y
> 
> There are no error log entries regarding this user at all.
> 
> After some research I tried to start the replication manually with
> "doveadm -D backup -u mail at example.com -d tcp:x.y.z.11" and
finally
> I got an error message:
> 
> Jul 31 13:55:37
> doveadm(mail at example.com)<11341><rm1YCblt5mJNLAAAJFg/Tg>:
Debug: brain
> M: Mailbox INBOX: local=e74590221b6ce6620d29000024583f4e/0/1,
> remote=20a9ce2b1c6ce662244e0000baba0ddd/0/1: GUIDs conflict - will be
> merged later
> 
> As a matter of fact, the mails in the inbox are the ones that are
> missing on backend2.
> 
> 
> I always end up with this error, no matter what I do:
> - I completely removed the folder structure on backend2.
> - I removed all database entries.
> - I removed this user from the replication, ran imapsync again followed
> by "doveadm -D backup ..."
> 
> I also removed fts-flatcurve to rule this out as a cause of error.
> 
> I have no idea what to do. This configuration worked for months.
> I got this error with 2.3.17.1 and 2.3.19.1
> 
> Any hints would be highly appreciated.
> 
> 
> Regards
> Patrick
> 
> 
> 
> 
> 
> # 2.3.19.1 (9b53102964): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.19 (4eae2f79)
> auth_master_user_separator = *
> auth_mechanisms = plain login
> doveadm_password = # hidden, use -P to show it
> doveadm_port = 12345
> listen = x.y.z.12
> lmtp_save_to_detail_mailbox = yes
> log_debug = category=fts-flatcurve
> log_path = /var/log/dovecot.log
> mail_debug = yes
> mail_home = /srv/mail/%Ld/%Ln
> mail_location = maildir:~/Maildir
> mail_plugins = quota notify replication
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart
> extracttext editheader
> metric imapc_traffic {
>    fields = bytes_in bytes_out
>    filter = event=imap_command_finished
>    group_by = user
> }
> namespace inbox {
>    inbox = yes
>    location >    mailbox Drafts {
>      auto = subscribe
>      special_use = \Drafts
>    }
>    mailbox Junk {
>      special_use = \Junk
>    }
>    mailbox Sent {
>      auto = subscribe
>      special_use = \Sent
>    }
>    mailbox "Sent Messages" {
>      special_use = \Sent
>    }
>    mailbox Spamverdacht {
>      auto = subscribe
>      special_use = \Junk
>    }
>    mailbox Trash {
>      auto = subscribe
>      special_use = \Trash
>    }
>    prefix = INBOX/
>    separator = /
>    type = private
> }
> passdb {
>    args = /usr/local/etc/dovecot/master-user
>    driver = passwd-file
>    master = yes
>    pass = yes
> }
> passdb {
>    args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>    driver = sql
> }
> plugin {
>    mail_replica = tcp:x.y.z.11
>    quota = count:User quota
>    quota_grace = 25M
>    quota_rule2 = INBOX/Trash:storage=+100M
>    quota_status_nouser = DUNNO
>    quota_status_success = DUNNO
>    quota_vsizes = yes
>    quota_warning = storage=100%% quota-full %u
>    quota_warning2 = storage=95%% quota-warning 95 %u
>    quota_warning3 = storage=80%% quota-warning 80 %u
>    quota_warning4 = -storage=100%% quota-ok %u
>    sieve = ~/.dovecot.sieve
>    sieve_after = /usr/local/etc/dovecot/sieve/sieve_after.sieve
>    sieve_default = /usr/local/etc/dovecot/sieve/default.sieve
>    sieve_dir = ~/sieve
>    sieve_extensions = +editheader
> }
> protocols = imap pop3 lmtp sieve
> service aggregator {
>    fifo_listener replication-notify-fifo {
>      user = vmail
>    }
>    unix_listener replication-notify {
>      user = vmail
>    }
> }
> service auth {
>    unix_listener auth-userdb {
>      group = vmail
>      mode = 0666
>      user = vmail
>    }
> }
> service config {
>    unix_listener config {
>      mode = 0600
>      user = vmail
>    }
> }
> service doveadm {
>    inet_listener {
>      port = 12345
>    }
> }
> service imap-login {
>    process_min_avail = 2
>    service_count = 0
> }
> service lmtp {
>    executable = lmtp -L
>    inet_listener lmtp {
>      address = x.y.z.12
>      port = 24
>    }
>    process_min_avail = 20
> }
> service managesieve-login {
>    inet_listener sieve {
>      port = 4190
>    }
> }
> service quota-full {
>    executable = script /usr/local/etc/dovecot/quota_full.sh
>    unix_listener quota-full {
>      user = vmail
>    }
>    user = root
> }
> service quota-ok {
>    executable = script /usr/local/etc/dovecot/quota_ok.sh
>    unix_listener quota-ok {
>      user = vmail
>    }
>    user = root
> }
> service quota-status {
>    client_limit = 1
>    executable = quota-status -p postfix
>    inet_listener {
>      port = 12340
>    }
> }
> service quota-warning {
>    executable = script /usr/local/etc/dovecot/quota_warning.sh
>    unix_listener quota-warning {
>      user = vmail
>    }
>    user = root
> }
> service replicator {
>    process_min_avail = 1
>    unix_listener replicator-doveadm {
>      mode = 0600
>      user = vmail
>    }
> }
> ssl = required
> ssl_cert = </etc/ssl/certs/bundle.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>    driver = prefetch
> }
> userdb {
>    args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>    driver = sql
> }
> verbose_proctitle = yes
> protocol lmtp {
>    info_log_path = /var/log/dovecot-lmtp.log
>    mail_plugins = quota notify replication sieve
> }
> protocol imap {
>    mail_max_userip_connections = 40
>    mail_plugins = quota notify replication imap_quota
> }
>

Patrick Westenberg

2022-Aug-01 09:15 UTC

head link

Replication not working - GUIDs conflict - will be merged later

Very interesting new insights:

When I use imapsync and let it synchronize mails from INBOX to
INBOX/testfolder, the automatic replication works fine.
All mails are synchronized between my two backends.


When I move the mails to the INBOX (doveadm move -u mail at example.com
INBOX mailbox INBOX/testfolder all), these mails are lost on the
replica! They are neither in INBOX, nor in INBOX/testfolder

Regards
Patrick

Patrick Westenberg

2022-Aug-28 10:00 UTC

head link

Replication not working - GUIDs conflict - will be merged later

Here are some logs, maybe related to my problem.



Aug 23 20:43:57 Panic:
doveadm(mail at example.com)<1355655><rdvZDu0fBWOHrxQAuroN3Q>: file
dsync-brain-mailbox.c: line 883 (dsync_brain_slave_recv_mailbox):
assertion failed: (memcmp(dsync_box->mailbox_guid,
local_dsync_box.mailbox_guid, sizeof(dsync_box->mailbox_guid)) == 0)
Aug 23 20:43:57 Error:
doveadm(mail at example.com)<1355655><rdvZDu0fBWOHrxQAuroN3Q>: Raw
backtrace: /usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f3c4c30e562] ->
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7f3c4c30e67e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x1022db)
[0x7f3c4c31b2db] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x102371)
[0x7f3c4c31b371] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x55589)
[0x7f3c4c26e589] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](dsync_brain_slave_recv_mailbox+0x2f8)
[0x56099f680618] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](dsync_brain_run+0x216) [0x56099f67de26] ->
dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](+0x4b483) [0x56099f67e483] -> dovecot/doveadm-server
[172.17.1.12 mail at example.com slave_recv_mailbox](+0x5ce1f)
[0x56099f68fe1f] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69)
[0x7f3c4c331509] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132)
[0x7f3c4c332bf2] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50)
[0x7f3c4c3315b0] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x40)
[0x7f3c4c331770] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](+0x2f335) [0x56099f662335] -> dovecot/doveadm-server
[172.17.1.12 mail at example.com slave_recv_mailbox](+0x31ba5)
[0x56099f664ba5] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](doveadm_cmd_ver2_to_mail_cmd_wrapper+0x2ca)
[0x56099f665c5a] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](doveadm_cmd_run_ver2+0x501) [0x56099f675bb1] ->
dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](+0x46ffa) [0x56099f679ffa] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69)
[0x7f3c4c331509] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132)
[0x7f3c4c332bf2] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50)
[0x7f3c4c3315b0] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x40)
[0x7f3c4c331770] ->
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7f3c4c2a4333] -> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](main+0x102) [0x56099f655162] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f3c4bf30d0a]
-> dovecot/doveadm-server [172.17.1.12 mail at example.com
slave_recv_mailbox](_start+0x2a) [0x56099f6551ea]

dovecot - Jul 2022 - Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later

Replication not working - GUIDs conflict - will be merged later