dovecot - Jan 2020 - Upgrade 2.2.27 to 2.3.9.2: master(imap): net_connect_unix(imap) failed: Resource temporarily unavailable

On 07 Jan 2020, at 07:15, Dovecot Mailing List <dovecot at dovecot.org>
wrote:
>  dovecot: master: Warning: service(imap-login): process_limit (16) reached,
client connections are being dropped
Is this an actual problem? Most client connections are idle, so I doubt if the
occasional drop is noticed at all.

Also, why a limit of 16? Have you tried, oh, 160 instead?





-- 
"Master, what is the difference between a humanistic, monastic system
	of belief in which wisdom is sought by means of an apparently
	nonsensical system of questions and answers, and a lot of mystic
	gibberish made up on the spur of the moment?" Wen considered this
	for some time, and a last said: "A fish!?

On 08/01/2020 01:47, @lbutlr wrote:
> On 07 Jan 2020, at 07:15, Dovecot Mailing List <dovecot at
dovecot.org> wrote:
>>   dovecot: master: Warning: service(imap-login): process_limit (16)
reached, client connections are being dropped
> 
> Is this an actual problem? Most client connections are idle, so I doubt if
the occasional drop is noticed at all.
I assume it would mean Dovecot is dropping new incoming connections 
(e.g. new users trying to log in), not existing idle connections? 
Dropping new incoming connections is a major problem, because some email 
clients (e.g. Outlook) are notorious for giving the users password 
prompts whenever something unexpected happens.

It is a problem because it is unexpected behaviour. Where are the 
imap-login sockets going, if not to serve active TCP connections? On all 
servers, the number total-available imap-login processes is 
significantly larger than the number of TCP connections on the server. 
In some cases, it is about

~# ss -nt | wc -l
7817
~# ss -nt "( sport = :143 or sport = :993 )" | wc -l
7812
~# doveadm process status | grep "^imap-login "
imap-login 17384 1462            44553       0          1578475790 
   0
imap-login 17383 550             66830       0          1578475790 
   0
imap-login 17382 488             59074       0          1578475789 
   0
imap-login 17381 929             58251       0          1578475790 
   0
imap-login 17380 712             62790       0          1578475790 
   0
imap-login 17377 1608            35143       0          1578475790 
   0
imap-login 17376 1718            27955       0          1578475790 
   0
imap-login 17375 1880            14903       0          1578475789 
   0
imap-login 17374 1965            10754       0          1578475790 
   0
imap-login 17373 1796            21137       0          1578475790 
   0
imap-login 17372 1194            52360       0          1578475789 
   0
imap-login 17371 1994            7482        0          1578475790 
   0
imap-login 17370 2061            3793        0          1578475789 
   0
imap-login 17369 2127            2992        0          1578475789 
   0
imap-login 17368 2014            5152        0          1578475790 
   0
imap-login 17362 2144            2723        0          1578475787 
   0
~# doveadm process status | grep "^imap-login " | awk "{sum +=
\$3} END
{print 40000-sum}"
15385

Why is there a (growing) discrepancy between the number of active TCP 
connections, and the number of reported free imap-login sockets? Is 
something else using imap-login thr
> Also, why a limit of 16? Have you tried, oh, 160 instead?
The limit was set to match the process_limit of the imap service 
(16*1250=20000).

We have made the assumption that imap-login should not require more 
sockets than the maximum number of concurrent imap sessions. We have not 
had any issues with this before upgrading, but it appears to not be the 
case in 2.3.x?


-- 
Eirik

On 08/01/2020 10:49, Eirik Rye wrote:
> [...]
In any case, the biggest issue is the one described in the original 
message in this thread, where logins are periodically failing for many 
for users:

2020-01-08T10:54:57.436400+01:00 imap dovecot: imap-login: Error: 
master(imap): net_connect_unix(imap) failed: Resource temporarily 
unavailable - http://wiki2.dovecot.org/SocketUnavailable
2020-01-08T10:54:57.437550+01:00 imap dovecot: imap-login: Internal 
login failure

At the same time:

~# ps aux | grep "dovecot/imap$" | wc -l
6637
~# doveadm process status | grep "^imap " | wc -l
6660
~# cat /proc/`pidof dovecot`/limits | grep "open files"
Max open files            32768                32768                files
~# doveconf -n
[...]
service imap-login {
   client_limit = 2500
   inet_listener imap {
     port = 143
   }
   inet_listener imaps {
     port = 993
     ssl = yes
   }
   process_limit = 16
   process_min_avail = 16
   service_count = 0
   vsz_limit = 512 M
}
service imap {
   client_limit = 1
   process_limit = 20000
}

-- 
Eirik