Jasper Siepkes
2020-Jan-26 13:32 UTC
Number of imap-login processes always keeps growing, never goes down
Hi all!
I've bumped into an issue which Dovecot which has me a bit stumped; All of a
sudden (ie. no obvious changes made to the config as far as I know) Dovecot
seems to never stop 'imap-login' processes. This continues to the point
where
it will hit the 'process_limit'. For a 50-ish user install the limit was
set to
100 but I increased it to 512. The only thing this changes it that Dovecot
takes a little longer to hit the limit.
I started having this problem with Dovecot version 2.3.2.1. After which I
updated to the latest version of Dovecot (2.3.9.2) to see if that would fix my
problem. However I'm still experiencing the same issue.
For a bit of context; We use Dovecot with Open-Xchange where users login via
OIDC, get a token and then use that token with Dovecot with the
'oauthbearer'
auth method. However users can also login via a username / password combo stored
in LDAP for clients that don't support 'oauthbearer' (ie. about all
the mail
clients ;-). We run Dovecot on SmartOS (ie. Illumos, a Solaris derivative).
Oddly enough 'doveadm' is under the impression there are only 3 users
loggedin
(when there are 512 imap-login processes). I think 'doveadm who'
doesn't show
users who are authenticated via an token (oauthbearer) because I only see 3
users and I don't see my own user which is logged in via Open-Xchange by
using
oauthbearer as auth method. I've anonimized the output but you get the idea:
----8<--------------------
# doveadm who
username # proto (pids) (ips)
foo1 at foo.nl 2 imap (78393 78391) (52.XXX.XXX.XXX)
foo2 at foo.nl 2 imap (72548 72547) (52.XXX.XXX.XXX)
foo3 at foo.nl 2 imap (480893 481231) (54.XXX.XXX.XXX 54.XXX.XXX.XXX)
----8<--------------------
I don't know if the above issue is related to my problem but the oauthbearer
sessions not showing up seems like a bug? Meanwhile there are 512 Dovecot
imap-login processes:
----8<--------------------
# ptree | grep imap-login | wc -l
513
----8<--------------------
I realise that the 'imap-login' process also works as a IMAP proxy and
that
it stays alive during the entire session of the client since it handles the TLS
part of the connection. However there are nowhere near 512 connections active.
As
far as I know IMAP doesn't support multiplexing (ie. multiple IMAP sessions
in a
single TCP connection) so with 512 imap-login proccess I would expect to see
a lot more connections then I'm seeing:
----8<--------------------
# netstat -a
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ------ ------ ------ ------
-----------
*.ssh *.* 0 0 1048576 0 LISTEN
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 52.XXX.XXX.XXX.53136
178176 0 1049580 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 52.XXX.XXX.XXX.53188
126848 0 1049580 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 83.XXX.XXX.XXX.64845
132352 0 1048960 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.2.7937
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.3.4983
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.2.34369
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.3.15041
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.2.49044
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.3.6340
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 10.100.2.2.11331
16384 0 1049800 0 CLOSE_WAIT
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 52.XXX.XXX.XXX.49920
94976 0 1049580 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.58569 10.100.3.84.ldap
1049792 0 1049800 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.33749 10.100.3.84.ldap
1049792 0 1049800 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps 52.XXX.XXX.XXX.49966
46464 0 1049580 0 ESTABLISHED
*.4190 *.* 0 0 1048576 0 LISTEN
*.24 *.* 0 0 1048576 0 LISTEN
*.imap2 *.* 0 0 1048576 0 LISTEN
*.imaps *.* 0 0 1048576 0 LISTEN
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps
ec2-54-210-254-232.compute-1.amazonaws.com.32892 107904 0 1049800 0
ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.ssh 10.101.2.14.59256
64128 35 1049880 0 ESTABLISHED
dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl.imaps
ec2-54-167-34-137.compute-1.amazonaws.com.3890 549632 0 1049800 0
ESTABLISHED
TCP: IPv6
Local Address Remote Address Swind
Send-Q Rwind Recv-Q State If
--------------------------------- --------------------------------- ------
------ ------ ------ ----------- -----
*.ssh *.* 0
0 1048576 0 LISTEN
Active UNIX domain sockets
Address Type Vnode Conn Local Address
Remote Address
----8<--------------------
This is the running Dovecot config:
----8<--------------------
# doveconf -n -c /etc/dovecot/dovecot.conf
# 2.3.9.2 (cf2918cac): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.9 (db4e9a2f)
# OS: SunOS 5.11 i86pc
# Hostname: dovecot-1.inst.sp-prod.nl1.cns.supersecretcorp.nl
auth_failure_delay = 3 secs
auth_mechanisms = plain oauthbearer xoauth2
base_dir = /var/run/dovecot/
disable_plaintext_auth = no
first_valid_uid = 1000
hostname = imap.supersecretcorp.nl
instance_name = dovecot-1
listen = *
lmtp_rcpt_check_quota = yes
log_path = /dev/stderr
mail_attachment_dir = /var/lib/dovecot/attachments
mail_attachment_fs = sis-queue /var/lib/dovecot/attachments/queue:posix
mail_attachment_hash = %{sha256}
mail_attribute_dict = file:~/mdbox/dovecot-attributes
mail_gid = vmail
mail_home = /var/lib/dovecot/vmail/%d/%n
mail_location = mdbox:~/mdbox
mail_plugins = " quota notify"
mail_server_admin = mailto:it at ask.supersecretcorp.nl
mail_temp_dir = /var/lib/dovecot/tmp
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext
metric imap_command_fetch_ok {
event_name = imap_command_finished
filter {
name = FETCH
tagged_reply_stat = OK
}
}
metric imap_command_list {
event_name = imap_command_finished
filter {
name = LIST
tagged_reply_state = OK
}
}
metric imap_command_search {
event_name = imap_command_finished
filter {
name = SEARCH
tagged_reply_stat = OK
}
}
metric imap_command_select {
event_name = imap_command_finished
filter {
name = SELECT
tagged_reply_state = OK
}
}
metric imap_select_no {
event_name = imap_command_finished
filter {
name = SELECT
tagged_reply_state = NO
}
}
metric imap_select_no_notfound {
event_name = imap_command_finished
filter {
name = SELECT
tagged_reply = NO*Mailbox doesn't exist:*
}
}
metric storage_http_gets {
categories = storage
event_name = http_request_finished
filter {
method = get
}
}
namespace inbox {
hidden = no
inbox = yes
list = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox "Sent objects" {
special_use = \Sent
}
mailbox Spam {
special_use = \Junk
}
mailbox Trash {
special_use = \Trash
}
prefix separator = /
subscriptions = yes
type = private
}
passdb {
args = /etc/dovecot/dovecot-oauth2.conf.ext
driver = oauth2
mechanisms = oauthbearer xoauth2
}
passdb {
args = /etc/dovecot/dovecot-ldap-passdb.conf.ext
driver = ldap
}
plugin {
push_notification_driver =
ox:url=http://#hidden_use-P_to_show#@open-xchange.svc.sp-prod.nl1.cns.supersecretcorp.nl:8009/preliminary/http-notify/v1/notify
user_from_metadata
quota = count:User quota
quota_rule2 = Trash:storage=+25M
quota_vsizes = yes
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_max_actions = 32
sieve_max_redirects = 4
sieve_max_script_size = 1M
sieve_quota_max_scripts = 500
sieve_quota_max_storage = 10M
sieve_vacation_send_from_recipient = yes
}
postmaster_address = postmaster at supersecretcorp.nl
protocols = imap lmtp sieve
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener auth-userdb {
group = vmail
mode = 0770
user = dovecot
}
user = $default_internal_user
}
service imap-login {
process_limit = 512
process_min_avail = 1
service_count = 1
}
service imap {
process_limit = 512
}
service lmtp {
inet_listener lmtp {
address = *
port = 24
}
user = vmail
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 0
service_count = 1
}
service managesieve {
process_limit = 256
}
service stats {
unix_listener stats-reader {
group = vmail
mode = 0660
user = vmail
}
unix_listener stats-writer {
group = vmail
mode = 0660
user = vmail
}
}
ssl_client_ca_dir = /opt/local/etc/openssl/certs
ssl_dh = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
submission_host = postfix.svc.sp-prod.nl1.cns.supersecretcorp.nl:25
userdb {
args = /etc/dovecot/dovecot-ldap-userdb.conf.ext
driver = ldap
}
protocol imap {
mail_plugins = " quota notify imap_quota"
ssl_cert = </etc/dovecot/pki/imap.supersecretcorp.nl.crt
ssl_key = # hidden, use -P to show it
}
protocol submission {
ssl_cert = </etc/dovecot/pki/smtp.supersecretcorp.nl.crt
ssl_key = # hidden, use -P to show it
}
protocol lmtp {
mail_plugins = quota sieve notify push_notification
postmaster_address = postmaster at supersecretcorp.nl
}
remote 10.100.2.0/23 {
protocol imap {
imap_metadata = yes
}
}
----8<--------------------
Hoping anyone can offer any insights!
Kind regards,
Jasper
Possibly Parallel Threads
Wisdom of the Ancients
