thr3ads.net - dovecot - huge increase in storage activity afther dovecot upgrade [Nov 2018]

If this information is useful, please help other people find it:
Share via:

Adrian M

2018-Nov-14 10:47 UTC

huge increase in storage activity afther dovecot upgrade

Thanks, they are as in example, except for "mailbox_list_index = yes"
witch
is from https://wiki.dovecot.org/PerformanceTuning

On Wed, Nov 14, 2018 at 12:18 PM Aki Tuomi <aki.tuomi at open-xchange.com>
wrote:
> You should review https://wiki2.dovecot.org/NFS to see that the settings
> make sense.
>
> Aki
> On 14.11.2018 12.00, Adrian M wrote:
>
> Thank you !
> I was little concerned that the following settings are not in line with
> the new version:
>
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mail_fsync = always
> mailbox_list_index = yes
> maildir_stat_dirs = yes
> mmap_disable = yes
>
>
> On Wed, Nov 14, 2018 at 10:19 AM Aki Tuomi <aki.tuomi at
open-xchange.com>
> wrote:
>
>> It should eventually wind down once all the problems are fixed. Of
course
>> if it does not happen, you can always run force-resync for the problem
>> users.
>>
>> Aki
>> On 14.11.2018 10.08, Adrian M wrote:
>>
>> Hi,
>> we upgraded our servers from version 2.2.13 to 2.2.27. After the
upgrade
>> we notice a 10x increase in traffic with the nfs storage an errors like
>> this in the logfile:
>>
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache
>> file /.../dovecot.index.cache: invalid record size
>> Nov 12 09:48:16 mail dovecot: imap(...): Error:
>> unlink(/.../dovecot.index.cache) failed: No such file or directory (in
>> mail-cache.c:29)
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index cache
>> file /.../dovecot.index.cache: invalid record size
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Broken file
>> /.../dovecot-uidlist line 8: Invalid data:
>>
>> Is this normal ?
>> Will the activity wind down ?
>> Can we do something, like deleting the old dovecot.index* or
>> dovecot-uidlist files from maildirs, or doing an doveadm force-refresh
for
>> all inboxes ?
>>
>> Thank you !
>>
>> Here's my configuration:
>>
>> # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.4.16 (fed8554)
>> # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 nfs
>> auth_failure_delay = 15 secs
>> auth_mechanisms = plain login
>> auth_verbose = yes
>> auth_worker_max_count = 256
>> default_client_limit = 40000
>> default_process_limit = 512
>> dict {
>>   lastlogin = mysql:/etc/dovecot/mysql/dovecot-dict-lastlogin.conf
>>   quotadict = mysql:/etc/dovecot/mysql/dovecot-dict-quota.conf
>> }
>> disable_plaintext_auth = no
>> first_valid_uid = 100
>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l
pid=%p %c
>> mail_fsync = always
>> mail_location = maildir:*/home/virtual/*%d/%u
>> mail_max_userip_connections = 16
>> mail_nfs_index = yes
>> mail_nfs_storage = yes
>> mail_plugins = zlib quota mail_log notify
>> mail_privileged_group = mail
>> mailbox_list_index = yes
>> maildir_stat_dirs = yes
>> managesieve_notify_capability = mailto
>> managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric relational
regex imap4flags copy include variables body enotify environment mailbox date
ihave
>> mmap_disable = yes
>> namespace inbox {
>>   inbox = yes
>>   location >>   mailbox Drafts {
>>     special_use = \Drafts
>>   }
>>   mailbox Junk {
>>     special_use = \Junk
>>   }
>>   mailbox Sent {
>>     auto = subscribe
>>     special_use = \Sent
>>   }
>>   mailbox "Sent Messages" {
>>     special_use = \Sent
>>   }
>>   mailbox Spam {
>>     auto = subscribe
>>     special_use = \Junk
>>   }
>>   mailbox Trash {
>>     auto = subscribe
>>     special_use = \Trash
>>   }
>>   mailbox virtual/All {
>>     special_use = \All
>>   }
>>   prefix >> }
>> passdb {
>>   args = /etc/dovecot/mysql/dovecot-sql.conf
>>   driver = sql
>> }
>> plugin {
>>   last_login_dict = proxy::lastlogin
>>   last_login_key = last-login/%u
>>   mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
>>   mail_log_fields = uid box msgid size
>>   quota = dict:user::proxy::quotadict
>>   quota_rule2 = Trash:ignore
>>   quota_rule3 = Spam:ignore
>>   quota_status_nouser = DUNNO
>>   quota_status_overquota = 552 5.2.2 The email account that you tried
to reach is over quota
>>   quota_status_success = DUNNO
>>   quota_warning = storage=95%% quota-warning 95 %u
>>   quota_warning2 = storage=80%% quota-warning 80 %u
>>   sieve = ~/.dovecot.sieve
>>   sieve_before = /etc/dovecot/sieve/default.sieve
>>   sieve_dir = ~/sieve
>>   stats_refresh = 30 secs
>>   stats_track_cmds = yes
>>   trash = /etc/dovecot/dovecot-trash.conf
>>   zlib_save = gz
>>   zlib_save_level = 6
>> }
>> protocols = imap sieve pop3 lmtp
>> service anvil {
>>   client_limit = 20000
>>   unix_listener anvil-auth-penalty {
>>     mode = 00
>>   }
>> }
>> service auth {
>>   client_limit = 20000
>>   unix_listener /var/spool/postfix/private/auth {
>>     group = postfix
>>     mode = 0660
>>     user = postfix
>>   }
>>   unix_listener auth-master {
>>     group = mail
>>     mode = 0660
>>     user = vmail
>>   }
>>   user = root
>>   vsz_limit = 1 G
>> }
>> service dict {
>>   unix_listener dict {
>>     mode = 0660
>>     user = vmail
>>   }
>> }
>> service imap-login {
>>   client_limit = 4096
>>   executable = /usr/lib/dovecot/imap-login
>>   inet_listener imap {
>>     address = *
>>     port = 143
>>   }
>>   inet_listener imaps {
>>     address = *
>>     port = 993
>>   }
>>   process_limit = 4096
>>   process_min_avail = 24
>>   service_count = 100
>> }
>> service imap {
>>   executable = /usr/lib/dovecot/imap
>>   process_limit = 4096
>>   process_min_avail = 24
>>   vsz_limit = 1 G
>> }
>> service lmtp {
>>   process_min_avail = 24
>>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>>     group = postfix
>>     mode = 0600
>>     user = postfix
>>   }
>>   vsz_limit = 1 G
>> }
>> service managesieve-login {
>>   client_limit = 4096
>>   inet_listener sieve {
>>     port = 4190
>>   }
>>   process_limit = 4096
>>   process_min_avail = 24
>>   service_count = 1
>> }
>> service managesieve {
>>   process_limit = 4096
>>   vsz_limit = 1 G
>> }
>> service pop3-login {
>>   client_limit = 4096
>>   executable = /usr/lib/dovecot/pop3-login
>>   inet_listener pop3 {
>>     address = *
>>     port = 0
>>   }
>>   inet_listener pop3s {
>>     address = *
>>     port = 995
>>   }
>>   process_limit = 4096
>>   process_min_avail = 24
>>   service_count = 100
>> }
>> service pop3 {
>>   executable = /usr/lib/dovecot/pop3
>>   process_limit = 4096
>>   process_min_avail = 24
>>   vsz_limit = 1 G
>> }
>> service quota-status {
>>   client_limit = 4096
>>   executable = /usr/lib/dovecot/quota-status -p postfix
>>   process_min_avail = 24
>>   unix_listener /var/spool/postfix/private/quota-status {
>>     group = postfix
>>     mode = 0660
>>     user = postfix
>>   }
>> }
>> service quota-warning {
>>   executable = script /usr/local/bin/quota-warning.sh
>>   unix_listener quota-warning {
>>     mode = 0660
>>     user = vmail
>>   }
>>   user = vmail
>> }
>>
>> userdb {
>>   args = /etc/dovecot/mysql/dovecot-sql.conf
>>   driver = sql
>> }
>> protocol imap {
>>   imap_idle_notify_interval = 4 mins
>>   imap_max_line_length = 64 k
>>   mail_max_userip_connections = 16
>>   mail_plugins = zlib quota mail_log notify trash imap_quota
>> }
>> protocol pop3 {
>>   mail_max_userip_connections = 16
>>   mail_plugins = zlib quota mail_log notify
>>   pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
>>   pop3_fast_size_lookups = yes
>>   pop3_no_flag_updates = yes
>>   pop3_uidl_format = %08Xu%08Xv
>> }
>> protocol lda {
>>   auth_socket_path = /var/run/dovecot/auth-master
>>   mail_plugins = zlib quota mail_log notify sieve
>>   sendmail_path = /usr/lib/sendmail
>>   syslog_facility = mail
>> }
>> protocol sieve {
>>   mail_max_userip_connections = 16
>>   managesieve_max_compile_errors = 5
>> }
>> protocol lmtp {
>>   mail_plugins = zlib quota mail_log notify sieve
>>   syslog_facility = mail
>> }
>>
>>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20181114/cf23ac8a/attachment.html>

Adi Pircalabu

2018-Nov-15 04:23 UTC

head link

huge increase in storage activity afther dovecot upgrade

Are you connecting to the same mailbox over NFS from multiple IMAP 
servers? If not and, at any given time, any mailbox will be accessed 
from a single NFS client, try to "dupe" Dovecot into thinking it's
not
using NFS. We're running quite successfully such setup with NFSv3 over 
TCP, which turned out to be the fastest and most reliable throughout the 
years. Here are the mount options:
rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=1892,mountproto=tcp,local_lock=none
On the Dovecot side we're running with:
lock_method = dotlock
mail_fsync = never
mail_nfs_index = no
mail_nfs_storage = no
maildir_very_dirty_syncs = yes
mmap_disable = yes
protocol lda {
   mail_fsync = optimized
}
protocol lmtp {
   mail_fsync = optimized
}

Note: we're using Maildir and the usual "works for me(c), may not work 
for everyone" applies.

-- 
Adi Pircalabu

On 2018-11-14 21:47, Adrian M wrote:> Thanks, they are as in example, except for "mailbox_list_index =
yes"
> witch is from https://wiki.dovecot.org/PerformanceTuning
> 
> On Wed, Nov 14, 2018 at 12:18 PM Aki Tuomi
> <aki.tuomi at open-xchange.com> wrote:
> 
>> You should review https://wiki2.dovecot.org/NFS to see that the
>> settings make sense.
>> 
>> Aki
>> On 14.11.2018 12.00, Adrian M wrote:
>> 
>> Thank you !
>> I was little concerned that the following settings are not in line
>> with the new version:
>> 
>> mail_nfs_index = yes
>> mail_nfs_storage = yes
>> mail_fsync = always
>> mailbox_list_index = yes
>> maildir_stat_dirs = yes
>> mmap_disable = yes
>> 
>> On Wed, Nov 14, 2018 at 10:19 AM Aki Tuomi
>> <aki.tuomi at open-xchange.com> wrote:
>> 
>> It should eventually wind down once all the problems are fixed. Of
>> course if it does not happen, you can always run force-resync for
>> the problem users.
>> 
>> Aki
>> On 14.11.2018 10.08, Adrian M wrote:
>> 
>> Hi,
>> we upgraded our servers from version 2.2.13 to 2.2.27. After the
>> upgrade we notice a 10x increase in traffic with the nfs storage an
>> errors like this in the logfile:
>> 
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index
>> cache file /.../dovecot.index.cache: invalid record size
>> Nov 12 09:48:16 mail dovecot: imap(...): Error:
>> unlink(/.../dovecot.index.cache) failed: No such file or directory
>> (in mail-cache.c:29)
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Corrupted index
>> cache file /.../dovecot.index.cache: invalid record size
>> Nov 12 09:48:16 mail dovecot: imap(...): Error: Broken file
>> /.../dovecot-uidlist line 8: Invalid data:
>> 
>> Is this normal ?
>> 
>> Will the activity wind down ?
>> Can we do something, like deleting the old dovecot.index* or
>> dovecot-uidlist files from maildirs, or doing an doveadm
>> force-refresh for all inboxes ?
>> 
>> Thank you !
>> 
>> Here's my configuration:
>> 
>> # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.4.16 (fed8554)
>> # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 nfs
>> auth_failure_delay = 15 secs
>> auth_mechanisms = plain login
>> auth_verbose = yes
>> auth_worker_max_count = 256
>> default_client_limit = 40000
>> default_process_limit = 512
>> dict {
>> lastlogin = mysql:/etc/dovecot/mysql/dovecot-dict-lastlogin.conf
>> quotadict = mysql:/etc/dovecot/mysql/dovecot-dict-quota.conf
>> }
>> disable_plaintext_auth = no
>> first_valid_uid = 100
>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l
pid=%p
>> %c
>> mail_fsync = always
>> mail_location = maildir:_/home/virtual/_%d/%u
>> mail_max_userip_connections = 16
>> mail_nfs_index = yes
>> mail_nfs_storage = yes
>> mail_plugins = zlib quota mail_log notify
>> mail_privileged_group = mail
>> mailbox_list_index = yes
>> maildir_stat_dirs = yes
>> managesieve_notify_capability = mailto
>> managesieve_sieve_capability = fileinto reject envelope
>> encoded-character vacation subaddress comparator-i;ascii-numeric
>> relational regex imap4flags copy include variables body enotify
>> environment mailbox date ihave
>> mmap_disable = yes
>> namespace inbox {
>> inbox = yes
>> location >> mailbox Drafts {
>> special_use = \Drafts
>> }
>> mailbox Junk {
>> special_use = \Junk
>> }
>> mailbox Sent {
>> auto = subscribe
>> special_use = \Sent
>> }
>> mailbox "Sent Messages" {
>> special_use = \Sent
>> }
>> mailbox Spam {
>> auto = subscribe
>> special_use = \Junk
>> }
>> mailbox Trash {
>> auto = subscribe
>> special_use = \Trash
>> }
>> mailbox virtual/All {
>> special_use = \All
>> }
>> prefix >> }
>> passdb {
>> args = /etc/dovecot/mysql/dovecot-sql.conf
>> driver = sql
>> }
>> plugin {
>> last_login_dict = proxy::lastlogin
>> last_login_key = last-login/%u
>> mail_log_events = delete undelete expunge copy mailbox_delete
>> mailbox_rename
>> mail_log_fields = uid box msgid size
>> quota = dict:user::proxy::quotadict
>> quota_rule2 = Trash:ignore
>> quota_rule3 = Spam:ignore
>> quota_status_nouser = DUNNO
>> quota_status_overquota = 552 5.2.2 The email account that you
>> tried to reach is over quota
>> quota_status_success = DUNNO
>> quota_warning = storage=95%% quota-warning 95 %u
>> quota_warning2 = storage=80%% quota-warning 80 %u
>> sieve = ~/.dovecot.sieve
>> sieve_before = /etc/dovecot/sieve/default.sieve
>> sieve_dir = ~/sieve
>> stats_refresh = 30 secs
>> stats_track_cmds = yes
>> trash = /etc/dovecot/dovecot-trash.conf
>> zlib_save = gz
>> zlib_save_level = 6
>> }
>> protocols = imap sieve pop3 lmtp
>> service anvil {
>> client_limit = 20000
>> unix_listener anvil-auth-penalty {
>> mode = 00
>> }
>> }
>> service auth {
>> client_limit = 20000
>> unix_listener /var/spool/postfix/private/auth {
>> group = postfix
>> mode = 0660
>> user = postfix
>> }
>> unix_listener auth-master {
>> group = mail
>> mode = 0660
>> user = vmail
>> }
>> user = root
>> vsz_limit = 1 G
>> }
>> service dict {
>> unix_listener dict {
>> mode = 0660
>> user = vmail
>> }
>> }
>> service imap-login {
>> client_limit = 4096
>> executable = /usr/lib/dovecot/imap-login
>> inet_listener imap {
>> address = *
>> port = 143
>> }
>> inet_listener imaps {
>> address = *
>> port = 993
>> }
>> process_limit = 4096
>> process_min_avail = 24
>> service_count = 100
>> }
>> service imap {
>> executable = /usr/lib/dovecot/imap
>> process_limit = 4096
>> process_min_avail = 24
>> vsz_limit = 1 G
>> }
>> service lmtp {
>> process_min_avail = 24
>> unix_listener /var/spool/postfix/private/dovecot-lmtp {
>> group = postfix
>> mode = 0600
>> user = postfix
>> }
>> vsz_limit = 1 G
>> }
>> service managesieve-login {
>> client_limit = 4096
>> inet_listener sieve {
>> port = 4190
>> }
>> process_limit = 4096
>> process_min_avail = 24
>> service_count = 1
>> }
>> service managesieve {
>> process_limit = 4096
>> vsz_limit = 1 G
>> }
>> service pop3-login {
>> client_limit = 4096
>> executable = /usr/lib/dovecot/pop3-login
>> inet_listener pop3 {
>> address = *
>> port = 0
>> }
>> inet_listener pop3s {
>> address = *
>> port = 995
>> }
>> process_limit = 4096
>> process_min_avail = 24
>> service_count = 100
>> }
>> service pop3 {
>> executable = /usr/lib/dovecot/pop3
>> process_limit = 4096
>> process_min_avail = 24
>> vsz_limit = 1 G
>> }
>> service quota-status {
>> client_limit = 4096
>> executable = /usr/lib/dovecot/quota-status -p postfix
>> process_min_avail = 24
>> unix_listener /var/spool/postfix/private/quota-status {
>> group = postfix
>> mode = 0660
>> user = postfix
>> }
>> }
>> service quota-warning {
>> executable = script /usr/local/bin/quota-warning.sh
>> unix_listener quota-warning {
>> mode = 0660
>> user = vmail
>> }
>> user = vmail
>> }
>> 
>> userdb {
>> args = /etc/dovecot/mysql/dovecot-sql.conf
>> driver = sql
>> }
>> protocol imap {
>> imap_idle_notify_interval = 4 mins
>> imap_max_line_length = 64 k
>> mail_max_userip_connections = 16
>> mail_plugins = zlib quota mail_log notify trash imap_quota
>> }
>> protocol pop3 {
>> mail_max_userip_connections = 16
>> mail_plugins = zlib quota mail_log notify
>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
>> pop3_fast_size_lookups = yes
>> pop3_no_flag_updates = yes
>> pop3_uidl_format = %08Xu%08Xv
>> }
>> protocol lda {
>> auth_socket_path = /var/run/dovecot/auth-master
>> mail_plugins = zlib quota mail_log notify sieve
>> sendmail_path = /usr/lib/sendmail
>> syslog_facility = mail
>> }
>> protocol sieve {
>> mail_max_userip_connections = 16
>> managesieve_max_compile_errors = 5
>> }
>> protocol lmtp {
>> mail_plugins = zlib quota mail_log notify sieve
>> syslog_facility = mail
>> }

Adrian Minta

2018-Nov-15 20:24 UTC

head link

huge increase in storage activity afther dovecot upgrade

Yes, multiple imap servers using one shared nfs storage. With the same 
config on 2.2.13 the public interface traffic was similar to the storage 
interface, around 100 mbps.

After we switch to 2.2.27 the storage interface traffic jumped 10 times 
while the public interface stayed the same. This make us thinking that 
something is wrong and each time a user logs in the whole Inbox content 
is read by dovecot.

What you are suggesting goes against the documentations and it may not 
be save, but I will give a? thought.

On 11/15/18 6:23 AM, Adi Pircalabu wrote:> Are you connecting to the same mailbox over NFS from multiple IMAP 
> servers? If not and, at any given time, any mailbox will be accessed 
> from a single NFS client, try to "dupe" Dovecot into thinking
it's not
> using NFS. We're running quite successfully such setup with NFSv3 over 
> TCP, which turned out to be the fastest and most reliable throughout 
> the years. Here are the mount options:
>
rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=1892,mountproto=tcp,local_lock=none
>
> On the Dovecot side we're running with:
> lock_method = dotlock
> mail_fsync = never
> mail_nfs_index = no
> mail_nfs_storage = no
> maildir_very_dirty_syncs = yes
> mmap_disable = yes
> protocol lda {
> ? mail_fsync = optimized
> }
> protocol lmtp {
> ? mail_fsync = optimized
> }
>
> Note: we're using Maildir and the usual "works for me(c), may not
work
> for everyone" applies.
>-- 
Best regards,
Adrian Minta

Possibly Parallel Threads

Search for more reasonably related threads

dovecot - Nov 2018 - huge increase in storage activity afther dovecot upgrade

huge increase in storage activity afther dovecot upgrade

huge increase in storage activity afther dovecot upgrade

huge increase in storage activity afther dovecot upgrade

Possibly Parallel Threads