dovecot - Nov 2017 - Replication oddities - different sizes between replicated nodes

Dear Dovecot community,

As discussing on IRC with Aki, I have the following thing happening:

I got an alert from my mailservice last night that I was running over quota. My
quota reached almost 91% (coming from 30%) in one day.
I do not recall receiving so much email (we talk about multiple gb?s additional
space usage), so something was acting up.

When I logged into my mailservers (I?ll describe my setup later on), it
mentioned that my FreeBSD mailbox was reaching 12gb of space usage.
While the seeming replicated machine (named A) only had 100mb of space in use
(which, with the amount of messages in the various mailboxes
is consistent with what I would expect).

I found out that the ?Junk? mailbox was the ?major' issue. I vaporized the
replicated (12GB) mailbox and issued a ?doveadm sync -u $user -N -U tcps:hostB`
so that I would expect the hostB to reach around 100mb-ish of email. I let it go
for a little and saw that the mail was done. When I issued a simple ?df -h? on
the directory, it was already multiple 100mb?s large and growing. It grew back
to multiple GB in little time. Again the Junk mailbox was the culprit. In the
mean
time I noticed my mailclient downloading 44k mails in that mailbox (where there
were only a few in the Junk mailbox normally).

I again vaporized the entire mailbox on hostB and cleaned out Junk and issued a
manual synchronisation. It again grew quickly.

Aki did an investigation with me and noticed that my rspamc script was not very
well written. I rewrote it as suggested (As written on the site).

This morning I continued with the investigation and it was again large on hostB
where it was still 100mb-ish on hostA. I vaporized host B again and
issued the manual sync with -Dv included per request of Aki. After letting it
run for a little hostA grew to 300mb and hostB restarted to reach 1.2GB and
remained static there. After half an hour or so the mailbox on hostA grew to
900mb and stayed there.

I investigated the mailboxes and saw 8 times the same email. Cleaning them up
with doveadm deduplicate -u $user mailbox ?*? made sure a lot of email
was cut from both hosts. I am now in the state where:

hostA: 268MB
hostB: 579MB

Both are for the same mailbox (I suspect that other mailboxes have similiar
issues).
When I look at the amount of messages counted by Apple Mail in the particular
mailbox I see 1399 emails. The amount of ?u? files is 14722 on hostA and 23590
on hostB.

I do not understand the difference from my mail client and the amount of u
files.

from mutt: Mutt: =mailbox [Msgs:1415 New:1 Flag:1  24M]-, which is roughly the
same as Apple mail reports.

I am a bit lost on where to look next, I would suspect that the synchronised
mailboxes would be similar in size, but not the one 2 as large, or before that
it grew and grew
and grew.

My personal feeling is that the sync takes place, but somehow it is not
registering that a message had been synchronised and does that again, and again,
and again till
the message gets synchronised right and the acknowledge is received and it stays
the same. That would at least support the 8times the same email that I saw at
first
and might also support the roughly double as big size of the other host (one
sync goes wrong, the next sync went right and got acked). At the bottom I also
referenced
two previous emails of mine which report similar oddities. Messages that were
removed on hostA got back within seconds as ?new message? . My feeling there is
that
the hostB has the ?failed sync copy with a different uid/guid? and tells hostA
that there is a new message and sends it over.

My doveconf -n for hostA (hostB follows), I did sanitize a bit wrt. IP addresses
and hostnames:

 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: FreeBSD 11.1-RELEASE amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
%k
mail_debug = yes
mail_fsync = always
mail_location = sdbox:~/sdbox
mail_plugins = " quota notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace {
  inbox = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Spam {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix   separator = .
}
passdb {
  driver = pam
}
plugin {
  imapsieve_mailbox1_before =
file:/usr/local/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_name = *
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  mail_replica = tcps:hostB:12346
  sieve = ~/.dovecot.sieve
  sieve_after = /usr/local/etc/dovecot/sieve/after/
  sieve_before = /usr/local/etc/dovecot/sieve/global/
  sieve_dir = ~/sieve
  sieve_execute_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_global = /usr/local/etc/dovecot/sieve/global/
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
  sieve_global_path = /usr/local/etc/dovecot/sieve/global/dovecot.sieve
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_vacation_dont_check_recipient = yes
}
postmaster_address = postmaster at isp
protocols = imap pop3 lmtp sieve
service aggregator {
  fifo_listener replication-notify-fifo {
    mode = 0666
  }
  unix_listener replication-notify {
    mode = 0666
  }
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
  }
}
service doveadm {
  inet_listener {
    port = 12346
    ssl = yes
  }
}
service imap-login {
  service_count = 1
}
service imap {
  process_limit = 1024
}
service lmtp {
  inet_listener lmtp {
    address = external ipv4 and ipv6 127.0.0.1 ::1
    port = 24
  }
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
service pop3 {
  process_limit = 1024
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0666
  }
}
ssl_ca = </usr/local/etc/dehydrated/certs/hostA/fullchain.pem
ssl_cert = </usr/local/etc/dehydrated/certs/hostA/cert.pem
ssl_client_ca_file = /usr/local/certificates/letsencrypt-ca.pem
ssl_dh_parameters_length = 2048
ssl_key =  # hidden, use -P to show it
ssl_protocols = !SSLv2 !SSLv3 TLSv1 TLSv1.1 TLSv1.2
userdb {
  driver = passwd
}
verbose_proctitle = yes
protocol lmtp {
  auth_username_format = %n
  mail_fsync = optimized
  mail_plugins = " quota notify replication sieve"
  postmaster_address = postmaster at isp
}
protocol lda {
  mail_plugins = " quota notify replication sieve"
}
protocol imap {
  imap_client_workarounds = delay-newmail
  mail_max_userip_connections = 50
  mail_plugins = " quota notify replication imap_quota imap_sieve"
}

for hostB:

# 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: FreeBSD 11.1-RELEASE-p1 amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
%k
mail_debug = yes
mail_fsync = always
mail_location = sdbox:~/sdbox
mail_plugins = " quota notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace {
  inbox = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Spam {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix   separator = .
}
passdb {
  driver = pam
}
plugin {
  imapsieve_mailbox1_before =
file:/usr/local/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_name = *
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  mail_replica = tcps:hostA:12346
  sieve = ~/.dovecot.sieve
  sieve_after = /usr/local/etc/dovecot/sieve/after/
  sieve_before = /usr/local/etc/dovecot/sieve/global/
  sieve_dir = ~/sieve
  sieve_execute_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_global = /usr/local/etc/dovecot/sieve/global/
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
  sieve_global_path = /usr/local/etc/dovecot/sieve/global/dovecot.sieve
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_vacation_dont_check_recipient = yes
}
postmaster_address = postmaster at isp
protocols = imap pop3 lmtp sieve
service aggregator {
  fifo_listener replication-notify-fifo {
    mode = 0666
  }
  unix_listener replication-notify {
    mode = 0666
  }
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
  }
}
service doveadm {
  inet_listener {
    port = 12346
    ssl = yes
  }
}
service imap-login {
  service_count = 1
}
service imap {
  process_limit = 1024
}
service lmtp {
  inet_listener lmtp {
    address = ipv4 and ipv6 address 127.0.0.1 ::1
    port = 24
  }
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
service pop3 {
  process_limit = 1024
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0666
  }
}
ssl_ca = </usr/local/etc/dehydrated/certs/hostA/fullchain.pem
ssl_cert = </usr/local/etc/dehydrated/certs/hostA/cert.pem
ssl_client_ca_file = /usr/local/certificates/letsencrypt-ca.pem
ssl_dh_parameters_length = 2048
ssl_key =  # hidden, use -P to show it
ssl_protocols = !SSLv2 !SSLv3 TLSv1 TLSv1.1 TLSv1.2
userdb {
  driver = passwd
}
verbose_proctitle = yes
protocol lmtp {
  auth_username_format = %n
  mail_fsync = optimized
  mail_plugins = " quota notify replication sieve"
  postmaster_address = postmaster at isp
}
protocol lda {
  mail_plugins = " quota notify replication sieve"
}
protocol imap {
  imap_client_workarounds = delay-newmail
  mail_max_userip_connections = 50
  mail_plugins = " quota notify replication imap_quota imap_sieve?
}

Both machines run FreeBSD, with a ZFS backend. HostB has quota?s on the zfs
shares but none of them are full or almost full and have plenty of space left on
their directories (so there is no quota or disk full issue there)
Mail is arriving on both hosts by upstream MX?es. Most of my users use hostB as
their main host. I use hostA as my primary host.

Both machines are located at Hetzner in Germany. HostA is in DC6 and HostB is in
DC12. Not situated next to eachother but low latency links between them. Both
machines are connected to gbit uplinks and are not very highly
loaded. The machines consist of 2x2TB disk (mirrored with ZFS) and 8cores, 32GB
RAM.

There were warnings when manually doing the sync that the mailbox changed in
between and should be reissued, after which the mailboxes kept growing.

Previously I reported two messages here which might be the foundation of the
same issue:
https://dovecot.org/list/dovecot/2016-July/104873.html and
https://www.dovecot.org/list/dovecot/2017-March/107459.html


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20171110/fabb0be8/attachment.sig>