dovecot - Feb 2011 - dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable

We recently switched over mail servers to one running dovecot and postfix 
on RedHat Enterprise 6.  The mail store is NFS on a NetApp filer, the 
index files are on local disk.  We have about 6000 total active users, 
though not all of them access the system at the same time.

All goes well for a while, then we start seeing errors like this in the 
log:

dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily
unavailable
imap-login: Internal login failure (auth failed, 1 attempts)

And at this point, users can't log in.  I don't know what Resource
dovecot
is running out of.

The machine is a 12-core Xeon 2.27Ghz, with 24GB RAM.

I have removed the 1024 nproc limit in 
/etc/security/limits.d/90-nproc.conf, and upped the nproc and nofile 
limits to 16384 and 65535 respectively, both in limits.conf, as well as 
adding ulimit statements in /etc/init.d/dovecot (belt and suspenders).

I have tried adjusting dovecot.conf, but I'm obviously missing something.

dovecot -n follows:

# 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-71.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux 
Server release 6.0 (Santiago)
auth_master_user_separator = *
auth_mechanisms = plain login
default_client_limit = 5000
default_process_limit = 5000
disable_plaintext_auth = no
mail_fsync = always
mail_gid = 501
mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n
mail_nfs_storage = yes
mail_plugins = quota
mail_uid = 501
passdb {
   args = /etc/dovecot/passwd.masterusers
   driver = passwd-file
   master = yes
   pass = yes
}
passdb {
   args = /etc/dovecot/dovecot-ldap.conf
   driver = ldap
}
plugin {
   quota = maildir:User quota
   quota_rule = *:storage=200M
}
service auth {
   client_limit = 32768
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   user = root
}
service imap-login {
   process_min_avail = 12
   service_count = 0
}
service pop3-login {
   service_count = 0
}
shutdown_clients = no
ssl_cert = </etc/pki/postfix/certs/nexus.pem
ssl_key = </etc/pki/postfix/private/nexus_key.pem
userdb {
   args = /etc/dovecot/dovecot-ldap.conf
   driver = ldap
}
protocol imap {
   mail_plugins = $mail_plugins imap_quota
}

Thanks,
-Kurt Hockenbury

* Kurt Hockenbury <khockenb at stevens.edu>:
> We recently switched over mail servers to one running dovecot and
> postfix on RedHat Enterprise 6.  The mail store is NFS on a NetApp
> filer, the index files are on local disk.  We have about 6000 total
> active users, though not all of them access the system at the same
> time.
That sounds nice.
> # 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
Is this really a 2.0 beta? 

-- 
Ralf Hildebrandt
  Gesch?ftsbereich IT | Abteilung Netzwerk
  Charit? - Universit?tsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt at charite.de | http://www.charite.de

On Tue, 2011-02-01 at 11:31 -0500, Kurt Hockenbury wrote:
> dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource
temporarily unavailable
> imap-login: Internal login failure (auth failed, 1 attempts)
I did a lot of fixes related to this and since I haven't heard of it
happening for a while now with latest 2.0.x versions, I think I managed
to fix it..
> # 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
Unfortunately RHEL6 was frozen early during v2.0 development before all
the bugs were fixed. RHEL6.1's Dovecot is hopefully a pretty good one.
Until then, I'm not sure what you can do to avoid the above problem,
other than upgrade to a newer v2.0. You could try if this helps:

service imap {
  process_min_avail = 12
}

Ok, we updated to dovecot 2.0.9, and we're still seeing the 
"net_connect_unix(imap) failed" problem happen around peak login time,
9-10am.

  dovecot  -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-71.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux 
Server release 6.0 (Santiago)
auth_master_user_separator = *
auth_mechanisms = plain login
default_client_limit = 5000
default_process_limit = 5000
disable_plaintext_auth = no
mail_fsync = always
mail_gid = 501
mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n
mail_nfs_storage = yes
mail_plugins = quota
mail_uid = 501
passdb {
   args = /etc/dovecot/passwd.masterusers
   driver = passwd-file
   master = yes
   pass = yes
}
passdb {
   args = /etc/dovecot/dovecot-ldap.conf
   driver = ldap
}
plugin {
   quota = maildir:User quota
   quota_rule = *:storage=200M
}
service anvil {
   client_limit = 15000
}
service auth {
   client_limit = 32768
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   user = root
}
service imap-login {
   process_min_avail = 12
   service_count = 0
}
service pop3-login {
   service_count = 0
}
shutdown_clients = no
ssl_cert = </etc/pki/postfix/certs/nexus.pem
ssl_key = </etc/pki/postfix/private/nexus_key.pem
userdb {
   args = /etc/dovecot/dovecot-ldap.conf
   driver = ldap
}
protocol imap {
   mail_plugins = quota imap_quota
}