dovecot - Oct 2012 - Dovecot stops to work - anvil problem

Hi all,

we have a problem about anvil, it seems that when we have a high load the
dovecot stops to work. Sometimes it is sufficient to make a dovecot
reload, but sometimes we have to restart it.

These are the lines related to anvil in the dovecot.log:

[root at secchia ~]# grep anvil /var/log/dovecot.log | more
Oct 26 11:13:55 anvil: Error: net_accept() failed: Too many open files
Oct 26 11:14:32 imap-login: Error: net_connect_unix(anvil) failed:
Resource temporarily unavailable
Oct 26 11:14:32 imap-login: Fatal: Couldn't connect to anvil
Oct 26 11:14:33 pop3-login: Error: net_connect_unix(anvil) failed:
Resource temporarily unavailable
Oct 26 11:14:33 pop3-login: Fatal: Couldn't connect to anvil
[...] (many lines like these)
Oct 26 12:01:10 pop3-login: Fatal: Couldn't connect to anvil
Oct 26 12:01:18 auth: Error: read(anvil-auth-penalty) failed: Connection
reset by peer
Oct 26 12:01:18 auth: Error: read(anvil-auth-penalty) failed: Connection
reset by peer
Oct 26 12:01:18 auth: Error: net_connect_unix(anvil-auth-penalty) failed:
Connection refused
Oct 26 12:01:18 auth: Error: net_connect_unix(anvil-auth-penalty) failed:
Connection refused
Oct 26 12:01:18 auth: Error: read(anvil-auth-penalty) failed: Connection
reset by peer
Oct 26 12:01:18 auth: Error: net_connect_unix(anvil-auth-penalty) failed:
Connection refused

And this is the output of the doveconf -n:

[root at secchia ~]# doveconf -n
# 2.0.1: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-308.11.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.8 (Tikanga) xfs
auth_cache_size = 1024
auth_cache_ttl = 21600 s
auth_debug = yes
auth_debug_passwords = yes
auth_master_user_separator = *
auth_mechanisms = plain login
auth_socket_path = /var/run/dovecot/auth-userdb
auth_verbose = yes
base_dir = /var/run/dovecot/
disable_plaintext_auth = no
hostname = mail.unimore.it
info_log_path = /var/log/dovecot.log
lda_mailbox_autocreate = yes
log_path = /var/log/dovecot.log
mail_debug = yes
mail_location = maildir:/cl/mail/vhosts/sms.unimo.it/%Ln/Maildir
mail_plugins = $mail_plugins quota
mailbox_idle_check_interval = 60 s
mbox_write_locks = fcntl
namespace {
  inbox = yes
  location   prefix = INBOX.
  separator = .
  type = private
}
passdb {
  args = /usr/local/etc/dovecot.masterusers
  driver = passwd-file
  master = yes
}
passdb {
  args = dovecot
  driver = pam
}
plugin {
  quota = maildir:User quota
  quota_exceeded_message = Quota exceeded (mailbox is full)
  quota_rule = *:storage=200MB
  quota_rule2 = *:messages=100000
  quota_rule3 = INBOX.Trash:storage=+100M
  quota_warning = storage=90%% quota-warning 90 %u
  quota_warning2 = storage=85%% quota-warning 85 %u
  quota_warning3 = messages=95%% quota-warning 95 %u
  quota_warning4 = messages=80%% quota-warning 80 %u
  setting_name = quota
}
postmaster_address = postmaster at unimore.it
quota_full_tempfail = yes
service anvil {
  client_limit = 199999
  process_limit = 199999
}
service auth {
  client_limit = 14500
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
  }
  process_limit = 5000
}
service imap {
  process_limit = 5000
}
service pop3-login {
  inet_listener pop3 {
    port = 110
  }
  inet_listener pop3s {
    port = 995
  }
}
service pop3 {
  process_limit = 1024
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  unix_listener quota-warning {
    group = vmail
    user = vmail
  }
  user = dovecot
}
ssl_ca = </etc/pki/tls/certs/ca_unimore_tcs.pem
ssl_cert = </etc/pki/tls/certs/cert-852-mail.unimore.it-cluster.pem
ssl_key = </etc/pki/tls/certs/mailcluster.key
userdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
verbose_ssl = yes
protocol lda {
  mail_plugins = $mail_plugins quota
}
protocol imap {
  mail_plugins = $mail_plugins imap_quota
}
protocol pop3 {
  mail_plugins = $mail_plugins quota
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = %08Xu%08Xv
}
[root at secchia ~]#

And these are the limit settings in the OS:
*               soft    nofile          131072
*               hard    nofile          131072

Have someone had the same problem?

thanks in advance

Fabio Ferrari

On 26.10.2012, at 13.24, FABIO FERRARI wrote:
> Hi all,
> 
> we have a problem about anvil, it seems that when we have a high load the
> dovecot stops to work. Sometimes it is sufficient to make a dovecot
> reload, but sometimes we have to restart it.
> 
> Oct 26 11:13:55 anvil: Error: net_accept() failed: Too many open files
This is the problem.
> And these are the limit settings in the OS:
> *               soft    nofile          131072
> *               hard    nofile          131072
> 
> Have someone had the same problem?
The OS limits are ok. But you need to make sure that the dovecot processes have
enough fds in ulimit. You can check the limits with:

cat /proc/<pid of anvil process>/limits

The "Max open files" soft limit is what you're most likely
hitting. Use "ulimit -n 10000" or something before running dovecot
binary. And make sure that it changes the limit in the proc. Many init scripts
change the ulimit internally.