dovecot - Feb 2012 - log message about client_limit may in fact be due to ulimit

Hi,

our Dovecot proxy started logging about reaching the client_limit once 
there were about 1000 connections running over it:

Feb 14 15:22:02 vmmailproxy1 dovecot: master: Warning: service(anvil): 
client_limit (100000) reached, client connections are being dropped
Feb 14 15:22:18 vmmailproxy1 dovecot: master: Warning: service(ipc): 
client_limit (100000) reached, client connections are being dropped

I was a bit perplexed because 100.000 is far above 1000.  However we 
found out that in fact the problem was the number of open files, and we 
have fixed it by putting 'ulimit -n 6000' into the init script.

Even though the log claims that connections would be dropped, the number 
of imap-login processes kept rising above 1000.  The "doveadm proxy 
list" would fail with an error message:

$ sudo doveadm proxy list
doveadm(root): Error: LIST failed: Disconnected
Segmentation fault

The segfault did not happen every time I tried it.  I'm sorry I don't 
have any core dumps or backtraces, because I was more concerned about 
putting the proxy into a sane state.  Maybe I can reproduce this on a 
test server if you would like a back trace.

Anyway, I suggest changing the log message if the failure is not 
actually due to the client_limit.


# 2.0.18: /etc/ports/dovecot/dovecot.conf
# OS: Linux 2.6.32-38-server x86_64 Ubuntu 10.04.2 LTS
auth_username_format = %n
default_client_limit = 60000
default_process_limit = 10000
disable_plaintext_auth = no
mail_location = maildir:/nonexistent
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave
passdb {
   args = /etc/ports/dovecot/passwd
   driver = passwd-file
}
protocols = imap pop3 lmtp sieve
service anvil {
   client_limit = 100000
}
service auth {
   client_limit = 60000
}
service ipc {
   client_limit = 100000
}
service managesieve-login {
   inet_listener sieve {
     port = 4190
   }
   inet_listener sieve_deprecated {
     port = 2000
   }
}
shutdown_clients = no
ssl_cert = </etc/ssl/certs/mailproxy+chain.pem
ssl_key = </etc/ssl/private/mailproxy.key
syslog_facility = local6
verbose_proctitle = yes
protocol imap {
   imap_capability = IMAP4rev1 IDLE THREAD=REFERENCES SORT QUOTA ACL UIDPLUS
}


The passwd file has all entries looking like this:
login:::::::proxy=y host=backend1 nopassword=yes


-- 
Christoph Bu?enius
Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik
Technische Universit?t M?nchen
+49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <>
Garching

On 14.2.2012, at 17.04, Christoph Bu?enius wrote:
> our Dovecot proxy started logging about reaching the client_limit once
there were about 1000 connections running over it:
> 
> Feb 14 15:22:02 vmmailproxy1 dovecot: master: Warning: service(anvil):
client_limit (100000) reached, client connections are being dropped
> Feb 14 15:22:18 vmmailproxy1 dovecot: master: Warning: service(ipc):
client_limit (100000) reached, client connections are being dropped
This indicate some kind of a connection leak.. There shouldn't be so many
connections. Each login process is supposed to create only one connection to
anvil and one connection to ipc. How many open fds do you see in those processes
currently?
> I was a bit perplexed because 100.000 is far above 1000.  However we found
out that in fact the problem was the number of open files, and we have fixed it
by putting 'ulimit -n 6000' into the init script.
The number of open files is so large, because there are at least 200k fds used
for connections to anvil and ipc processes. Do you see some error message also
about "Too many open files", or why do you think the number of open
files was the problem? If a process runs out of fds there should be such an
error message.
> Even though the log claims that connections would be dropped, the number of
imap-login processes kept rising above 1000.
Connections to anvil and ipc processes were being dropped. That doesn't
necessarily mean that everything else stops.
>  The "doveadm proxy list" would fail with an error message:
> 
> $ sudo doveadm proxy list
> doveadm(root): Error: LIST failed: Disconnected
> Segmentation fault
> 
> The segfault did not happen every time I tried it.  I'm sorry I
don't have any core dumps or backtraces, because I was more concerned about
putting the proxy into a sane state.  Maybe I can reproduce this on a test
server if you would like a back trace.
http://hg.dovecot.org/dovecot-2.0/rev/c402fb407501 fixes the crash I think.