We are gradually rolling out Dovecot (IMAP only, no POP3) to our
customer base. We are replicating between a pair of CentOS 7 boxes.
All has been working wonderfully. However, to be sure our rollout
continues to go smoothly, we put together a simple benchmark client
program to fire up X persistent IMAP connections (using hundreds of
mailboxes) that login, list the folders, select the INBOX, periodically
FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As
long as we keep the number of concurrent IMAP connections under 2k,
everything works fine. 2,000 processes generate a minimal load on the
server. However, 2048 is a "hard limit" for number of IMAP processes
on
a box the way things are configured right now. The number (2048) is a
bit too magical to be anything but a misconfiguration of either some
kernel limit or a dovecot configuration limit. I have been unable to
figure out where else to look. I have pasted in my dovecot config below
along with the kernel setting for max_user_instances.
Hoping someone can tell me what stupid mistake I have made or what else
to check.
Just an aside, we front-end this with a separate set of proxy servers
that provide an SSL front-end, hence, no SSL in the dovecot config.
/proc/sys/fs/inotify/max_user_instances = 4096
# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-123.13.1.el7.x86_64 x86_64 CentOS Linux release
7.0.1406 (Cor
e)
auth_master_user_separator = *
auth_mechanisms = plain login
default_internal_user = sysadm
default_login_user = sysadm
default_vsz_limit = 512 M
disable_plaintext_auth = no
doveadm_password = **************
doveadm_port = 12345
imap_max_line_length = 128 k
listen = *
mail_gid = 2001
mail_location = maildir:~/Maildir
mail_plugins = notify replication
mail_uid = 2001
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
mbox_write_locks = fcntl
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix }
passdb {
args = scheme=PLAIN /z/imap/usr/master-users
driver = passwd-file
master = yes
pass = yes
}
passdb {
args = scheme=PLAIN username_format=%u /z/imap/usr/users
driver = passwd-file
}
plugin {
mail_replica = tcp:imap0s
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_max_actions = 32
sieve_max_redirects = 4
sieve_max_script_size = 1M
}
protocols = imap pop3 sieve
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
user = sysadm
}
unix_listener replication-notify {
mode = 0666
user = sysadm
}
}
service auth {
unix_listener auth-userdb {
mode = 0777
}
}
service doveadm {
inet_listener {
port = 12345
}
user = sysadm
}
service imap {
process_limit = 4096
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service managesieve {
process_limit = 4096
}
service replicator {
process_min_avail = 1
}
ssl = no
userdb {
args = username_format=%u /z/imap/usr/users
driver = passwd-file
}
protocol lda {
mail_plugins = sieve notify replication
}
protocol imap {
mail_max_userip_connections = 40
}
protocol sieve {
managesieve_max_line_length = 65536
}
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 3 Feb 2015, Ron Cleven wrote:> We are gradually rolling out Dovecot (IMAP only, no POP3) to our customer > base. We are replicating between a pair of CentOS 7 boxes. All has been > working wonderfully. However, to be sure our rollout continues to go > smoothly, we put together a simple benchmark client program to fire up X > persistent IMAP connections (using hundreds of mailboxes) that login, list > the folders, select the INBOX, periodically FETCH an email from the INBOX, > and otherwise sit in an IDLE loop. As long as we keep the number of > concurrent IMAP connections under 2k, everything works fine. 2,000 processes > generate a minimal load on the server. However, 2048 is a "hard limit" for > number of IMAP processes on a box the way things are configured right now. > The number (2048) is a bit too magical to be anything but a misconfiguration > of either some kernel limit or a dovecot configuration limit. I have been > unable to figure out where else to look. I have pasted in my dovecot config > below along with the kernel setting for max_user_instances. > > Hoping someone can tell me what stupid mistake I have made or what else to > check. > > Just an aside, we front-end this with a separate set of proxy servers that > provide an SSL front-end, hence, no SSL in the dovecot config. > > /proc/sys/fs/inotify/max_user_instances = 4096Do you use virtual users? Then this setting is way too low. There should be entries in kernel log or Dovecot. - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEVAwUBVNDiOHz1H7kL/d9rAQKbjwf+IhLHWBSM7I/G9A5I/RmlpcvbClYdYDlk BrSPaUnKkNFC1Gxm5vzNNnQc1lBlpfhUT+BEQbeB2kUkM2khH4pOt5BJarRYrq8o BjcMrkqIUJuYa/kzPgXpgonhywXWCww5AAA2j2w0I/sjPcoxFcEHrPluPq3CasWx VnB5Y9OLzVeALbuGlmC+/hc+K4SJWdL314hGxC5RTNdTY4bHusF/cUJzv63At3fZ V7ahL77+QQ8elMCmRHFXiT2pCHzx6qw54ZUA6hIDt5VPgsMhrBaF8pwr2LIIHhsz 6VwvMvj5A+N8XH3z2QUVa36V5YvQrYoHVqaO/n+5E/LymcvBXx92QA==1IfD -----END PGP SIGNATURE-----
On 03.02.2015 15:35, Ron Cleven wrote:> /proc/sys/fs/inotify/max_user_instances = 4096Maybe you are hitting some file descriptor limit (kernel or ulimit)? This could help: https://unix.stackexchange.com/questions/36841/why-is-number-of-open-files-limited-in-linux -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20150203/a05310e5/attachment.sig>
On 02/03/2015 03:35 PM, Ron Cleven wrote:> We are gradually rolling out Dovecot (IMAP only, no POP3) to our > customer base. We are replicating between a pair of CentOS 7 boxes. All > has been working wonderfully. However, to be sure our rollout continues > to go smoothly, we put together a simple benchmark client program to > fire up X persistent IMAP connections (using hundreds of mailboxes) that > login, list the folders, select the INBOX, periodically FETCH an email > from the INBOX, and otherwise sit in an IDLE loop. As long as we keep > the number of concurrent IMAP connections under 2k, everything works > fine. 2,000 processes generate a minimal load on the server. However, > 2048 is a "hard limit" for number of IMAP processes on a box the way > things are configured right now. The number (2048) is a bit too magical > to be anything but a misconfiguration of either some kernel limit or a > dovecot configuration limit. I have been unable to figure out where > else to look. I have pasted in my dovecot config below along with the > kernel setting for max_user_instances. > > Hoping someone can tell me what stupid mistake I have made or what else > to check. > > Just an aside, we front-end this with a separate set of proxy servers > that provide an SSL front-end, hence, no SSL in the dovecot config. >You didn't mention but did you try checking process_limit and client_limit for various Dovecot services in the output of doveconf | less ? If I remember correctly, as long as imap process lives, it is a client of anvil process. When anvil reaches it's maximum, it's no longer possible to run more imap processes and therefore no longer possible to connect more clients. I think I hit some similar problem in the past, but in my case Dovecot complained during startup and warned that anvil's client_limit is too low compared to imap process_limit and that it could lead to problems. So I just increased the limit and was done with it. That said, this was some time ago so process and setting names can be completely wrong. Hope someone more knowledgeable corrects me in that case
The default connection limit values of dovecot are pathetic - unless you run a home mail server where there is only you, the cat and the dog using it. OP, lots of editing with trial and error required. from memory massive increases needed to: service auth -> client limit service imap-login -> process limit service imap -> process limit if using pop3, repeat above two steps s/imap/pop3/ service managesieve -> process limit service anvil -> client_limit (and system ulimit) I'm on holidays in australia at moment so don't have access to copy and show you a working example for 4K users, i will be back on hong kong in 17 days until then i have very limited access. On 2/4/15, Jiri Bourek <bourek at thinline.cz> wrote:> On 02/03/2015 03:35 PM, Ron Cleven wrote: >> We are gradually rolling out Dovecot (IMAP only, no POP3) to our >> customer base. We are replicating between a pair of CentOS 7 boxes. All >> has been working wonderfully. However, to be sure our rollout continues >> to go smoothly, we put together a simple benchmark client program to >> fire up X persistent IMAP connections (using hundreds of mailboxes) that >> login, list the folders, select the INBOX, periodically FETCH an email >> from the INBOX, and otherwise sit in an IDLE loop. As long as we keep >> the number of concurrent IMAP connections under 2k, everything works >> fine. 2,000 processes generate a minimal load on the server. However, >> 2048 is a "hard limit" for number of IMAP processes on a box the way >> things are configured right now. The number (2048) is a bit too magical >> to be anything but a misconfiguration of either some kernel limit or a >> dovecot configuration limit. I have been unable to figure out where >> else to look. I have pasted in my dovecot config below along with the >> kernel setting for max_user_instances. >> >> Hoping someone can tell me what stupid mistake I have made or what else >> to check. >> >> Just an aside, we front-end this with a separate set of proxy servers >> that provide an SSL front-end, hence, no SSL in the dovecot config. >> > > You didn't mention but did you try checking process_limit and > client_limit for various Dovecot services in the output of doveconf | > less ? If I remember correctly, as long as imap process lives, it is a > client of anvil process. When anvil reaches it's maximum, it's no longer > possible to run more imap processes and therefore no longer possible to > connect more clients. > > I think I hit some similar problem in the past, but in my case Dovecot > complained during startup and warned that anvil's client_limit is too > low compared to imap process_limit and that it could lead to problems. > So I just increased the limit and was done with it. > > That said, this was some time ago so process and setting names can be > completely wrong. Hope someone more knowledgeable corrects me in that case >