Dear list,
I have setup a master-master replication setup. My primairy MX's send email
over on a DNS loadbalanced way, so DNS is doing some kind of round-robin way of
sending mail to both master servers.
I found out, that on one of the two machines, the email synchronisation is
heavily delayed.
Lets assume server A receives a mail from the MX; it synchronises almost
instantly with the other server.
Whenever server B receives the email, it could take up to several hours to
synchronise the email, it seems that it is not detected prior.
It is also interesting to see, that the mailboxes on server A (Where users login
to retrieve their email via webmail/clients) are significantly smaller then the
mailboxes on server B. When investigating, it seems that "older"
mailboxes (or storage rather since we use mdbox) are still there on server B,
which already had been removed on server A.
My personal mailbox was 170MB on server A, while it was still 2.5GB on server B.
(which was around that size before cleaning up the mailsboxes).
I enabled debugging on the servers, and I see rather quick : "Replication
requests" on server A, but when getting an email on server B, I do not see
the request at all.
My servers are both running the same version, same configuration (utilizing
puppet), both running on ZFS and FreeBSD. Where server B is more loaded in
it's memory because of some bhyve VM's and the server A does not run any
VM.
Does someone have any pointers on where to look?
Thanks in advance ;-)
Remko
Included below the configurations from server A and B:
Server A:
# 2.2.25 (7be1766): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.14 (099a97c)
# OS: FreeBSD 10.3-RELEASE-p2 amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
haproxy_trusted_networks = YYYY
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
mail_debug = yes
mail_fsync = always
mail_location = mdbox:~/mdbox
mail_plugins = " quota notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = .
}
passdb {
driver = pam
}
plugin {
antispam_backend = mailtrain
antispam_mail_notspam = --ham
antispam_mail_sendmail = /usr/local/bin/sa-learn.sh
antispam_mail_spam = --spam
antispam_spam_pattern_ignorecase = spam;junk
antispam_trash_pattern_ignorecase = trash;deleted items;deleted messages
antispam_verbose_debug = 1
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
mail_log_fields = uid box msgid size
mail_replica = tcps:the other server:12346
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /usr/local/etc/dovecot/sieve/global/
sieve_global_path = /usr/local/etc/dovecot/sieve/default.sieve
}
postmaster_address = postmaster at xxx
protocols = imap pop3 lmtp sieve
replication_dsync_parameters = -d -N -l 60 -U
replication_max_conns = 100
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service doveadm {
inet_listener {
port = 12346
ssl = yes
}
}
service imap-login {
inet_listener imap_haproxy {
haproxy = yes
port = 10143
}
inet_listener imaps_haproxy {
haproxy = yes
port = 10144
ssl = yes
}
service_count = 1
}
service imap {
process_limit = 1024
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service pop3 {
process_limit = 1024
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_ca = </usr/local/etc/letsencrypt/live/XXX
ssl_cert = </usr/local/etc/letsencrypt/live/XXX
ssl_client_ca_file = /usr/local/certificates/letsencrypt-ca.pem
ssl_key = </usr/local/etc/letsencrypt/live/XXX
ssl_protocols = !SSLv2 !SSLv3
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol lmtp {
auth_username_format = %n
mail_plugins = quota sieve
postmaster_address = postmaster at xxxx
}
protocol lda {
mail_plugins = " quota notify replication sieve"
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = " quota notify replication imap_quota antispam"
}
Server B:
# 2.2.25 (7be1766): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.14 (099a97c)
# OS: FreeBSD 10.3-RELEASE amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
haproxy_trusted_networks = YYYY
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
mail_debug = yes
mail_fsync = always
mail_location = mdbox:~/mdbox
mail_plugins = " quota notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = .
}
passdb {
driver = pam
}
plugin {
antispam_backend = mailtrain
antispam_mail_notspam = --ham
antispam_mail_sendmail = /usr/local/bin/sa-learn.sh
antispam_mail_spam = --spam
antispam_spam_pattern_ignorecase = spam;junk
antispam_trash_pattern_ignorecase = trash;deleted items;deleted messages
antispam_verbose_debug = 1
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
mail_log_fields = uid box msgid size
mail_replica = tcps:the other host:12346
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /usr/local/etc/dovecot/sieve/global/
sieve_global_path = /usr/local/etc/dovecot/sieve/default.sieve
}
postmaster_address = postmaster at xxx
protocols = imap pop3 lmtp sieve
replication_dsync_parameters = -d -N -l 60 -U
replication_max_conns = 100
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service doveadm {
inet_listener {
port = 12346
ssl = yes
}
}
service imap-login {
inet_listener imap_haproxy {
haproxy = yes
port = 10143
}
inet_listener imaps_haproxy {
haproxy = yes
port = 10144
ssl = yes
}
service_count = 1
}
service imap {
process_limit = 1024
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service pop3 {
process_limit = 1024
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_ca = </usr/local/etc/letsencrypt/live/xxx
ssl_cert = </usr/local/etc/letsencrypt/live/xxx
ssl_client_ca_file = /usr/local/certificates/letsencrypt-ca.pem
ssl_key = </usr/local/etc/letsencrypt/live/xxx
ssl_protocols = !SSLv2 !SSLv3
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol lmtp {
auth_username_format = %n
mail_plugins = quota sieve
postmaster_address = postmaster at XXXXX
}
protocol lda {
mail_plugins = " quota notify replication sieve"
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = " quota notify replication imap_quota antispam"
}
You are not alone! On Wednesday, July 06, 2016 01:15:34 PM Remko Lodder wrote:> Dear list, > > I have setup a master-master replication setup. My primairy MX's send email > over on a DNS loadbalanced way, so DNS is doing some kind of round-robin > way of sending mail to both master servers. > > I found out, that on one of the two machines, the email synchronisation is > heavily delayed. Lets assume server A receives a mail from the MX; it > synchronises almost instantly with the other server. > > Whenever server B receives the email, it could take up to several hours to > synchronise the email, it seems that it is not detected prior.I have been dealing with this for months. http://www.dovecot.org/list/dovecot/2016-March/103680.html For a band aid I use this crontab entry. On the 2nd mail server. */15 * * * * root /usr/bin/doveadm sync -u "*" remote:mail1 However in doing this, and at other times during the sync. Something happens and kmail pulls in the email twice, and puts one in an odd state, grayed out in the GUI. I have to go do the directory and delete. Once read it has a T flag which other emails do not have. That becomes more common when I use the above, but otherwise happens on occasion.> It is also interesting to see, that the mailboxes on server A (Where users > login to retrieve their email via webmail/clients) are significantly > smaller then the mailboxes on server B. When investigating, it seems that > "older" mailboxes (or storage rather since we use mdbox) are still there on > server B, which already had been removed on server A.I experience every bit of what you are describing. Also seems to be effected when email arrives on one server, but users are checking/pulling email from another. They never see the ones on the other, and can have emails arrive, be deleted, etc.> My personal mailbox was 170MB on server A, while it was still 2.5GB on > server B. (which was around that size before cleaning up the mailsboxes). > > I enabled debugging on the servers, and I see rather quick : "Replication > requests" on server A, but when getting an email on server B, I do not see > the request at all. > > My servers are both running the same version, same configuration (utilizing > puppet), both running on ZFS and FreeBSD. Where server B is more loaded in > it's memory because of some bhyve VM's and the server A does not run any > VM. > > Does someone have any pointers on where to look?I have been hoping its some issue that gets fixed in some new release. It seems there might have been some regressions there, as at times it seemed to have gotten better and other times worse. I think it has something to do with full vs fast/quick syncing. I think the above command forces a full, and most times its doing a fast sync. There are not many settings to play with or adjust so seems to be something that requires addressing in the code itself unless some new settings are introduced. -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160711/8a6ee2a9/attachment.sig>
Quoting "William L. Thomson Jr." <wlt-ml at o-sinc.com>:> You are not alone! > > On Wednesday, July 06, 2016 01:15:34 PM Remko Lodder wrote: >> Dear list, >> >> I have setup a master-master replication setup. My primairy MX's send >> email >> over on a DNS loadbalanced way, so DNS is doing some kind of round-robin >> way of sending mail to both master servers. >> >> I found out, that on one of the two machines, the email synchronisation >> is >> heavily delayed. Lets assume server A receives a mail from the MX; it >> synchronises almost instantly with the other server. >> >> Whenever server B receives the email, it could take up to several hours >> to >> synchronise the email, it seems that it is not detected prior. > > I have been dealing with this for months. > http://www.dovecot.org/list/dovecot/2016-March/103680.html > > For a band aid I use? this crontab entry. On the 2nd mail server. > > */15 * * * *? ?root? ? /usr/bin/doveadm sync -u "*" remote:mail1 > <snip>Are you guys using LMTP to deliver from your MX server to the mailbox server? I have a similar setup, but not yet synched, because as I understand it - using 'deliver' to drop mail into an NFS mount won't inititate a sync.? I have to migrate my procmail scripts to sieve (and use the execute plugin) and change my final delivery to be a redirect to LMTP.?? Not sure how replication will work when running old procmail scripts from sieve... In any case..? If you're piping to dovecot's deliver/dovecot-lda, here is a rudimentary LMTP script I hacked together that I planned to use to replace deliver with...?? I'd grab the 'master' mailbox server IP for each user for the command line. #!/usr/bin/perl use Net::LMTP; use Getopt::Std; $opts{'s'} = "localhost"; $opts{'p'} = "24"; $opts{'f'} = 'root@' . `hostname`; chomp($opts{'f'}); chomp($opts{'s'}); getopts("hs:p:f:u:", \%opts); if ($opts{'h'}) { ??? print " lmtpsend [-s lmtpserver] [-f fromaddress] [-u subject] toaddress [...] ? lmtpsend will send an email from the commandline. ? Options: ??? -s lmtpserver????? Sets the lmtpserver for where to send the mail through. ??? -f fromaddress???? Sets the email address to be used on the From: line. ??? -u subject???????? Sets the email subject to be used from the Subject line. ??? toaddress????????? Where you want the email sent to. "; ??? exit; } die "no recepients to send mail to" if ($#ARGV < 0); @emailbody = <STDIN> ; # send the message $message = Net::LMTP->new($opts{'s'},$opts{'p'}) || die "can't talk to server $opts{'s'}\n"; $message->mail($opts{'f'}); $message->to(@ARGV) || die "failed to send to the recepients ",join(",", at ARGV),": $!"; $message->data(); $message->datasend("To: " . join(", ", at ARGV) . "\n"); $message->datasend(@emailbody); $message->dataend(); $message->quit; Rick
> On 11 Jul 2016, at 17:21, William L. Thomson Jr. <wlt-ml at o-sinc.com> wrote: > > You are not alone!Hello, Now that?s a relief! One of the things that I described and observed is that it seems that serverB is not seeing the email (or at least there is no connection that when an email is send and stored on the mailserver that the services see them and notify the other end). With tcpdump there is no traffic at all, until there is a sync the other way around. As said both systems are identical in hardware setup and use puppet to obtain their configuration, which is the same for both hosts (except the IP adresses and hostname); But since we are with at least two, we might have better luck in getting some help with this. I currently do not have an idea on where to look and how to investigate this properly. Any pointers from the list are welcome! Cheers Remko> > -- > William L. Thomson Jr. > Obsidian-Studios, Inc. > http://www.obsidian-studios.com