dovecot - Nov 2016 - auth client limit versus service count of mail processes

Hello,

We've got a pretty substantial dovecot installation (versions 2.1.7
and 2.2.13 on the backends, but tested with 2.2.24 as well) and this is the
relevant config snippet from 10-master.conf:

---
default_process_limit = 1024
default_client_limit = 16384

[...]

service imap {
  # Most of the memory goes to mmap()ing files. You may need to increase this
  # limit if you have huge mailboxes.
  #vsz_limit = $default_vsz_limit
  vsz_limit = 512M

  # Max. number of IMAP processes (connections)
  #process_limit = 1024
  process_limit = 524288
}

service pop3 {
  # Max. number of POP3 processes (connections)
  process_limit = 2048
  # Reduce spawns from hell
  service_count = 100
}
---

The above works fine, no warnings. 

Since 2 of our mailbox servers get a high number (2million/day) of IMAP
logins and resulting mail process spawns, I pondered doing the service
count bit for IMAP as well.

But adding a "service_count = 100" line (any value larger than 1
really) to
the imap section we get the dreaded: 
---
Nov 28 17:05:40 mbx09 dovecot: config: Warning: service auth {
client_limit=16384} is lower than required under max. load (528384)
---

And that's quite true, once reaching that limit auth will pile up and fail
eventually.

Clearly the pop3 part with a max of 2048 processes (we see about 300 at
peak times) neatly fits into the client limit of 16384.

Now setting the client limit in the auth section to the required value of
course will do no real good in this scenario, as the fd limit becomes the
next bottleneck and I'm not going to raise that to 500k, thank you very
much. ^o^

So basically my questions here are:

1. Where's the difference in Dovecot's logic between a mail service that
has a service count of 1 versus one with >1?

2. Any way to get the process recycling for IMAP going w/o setting the fd
limit to a ridiculous amount? 

Thanks,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/

On 29 Nov 2016, at 2.57, Christian Balzer <chibi at gol.com>
wrote:
> 
> service imap {
>  # Most of the memory goes to mmap()ing files. You may need to increase
this
>  # limit if you have huge mailboxes.
>  #vsz_limit = $default_vsz_limit
>  vsz_limit = 512M
> 
>  # Max. number of IMAP processes (connections)
>  #process_limit = 1024
>  process_limit = 524288
> }
..
> But adding a "service_count = 100" line (any value larger than 1
really) to
> the imap section we get the dreaded: 
> ---
> Nov 28 17:05:40 mbx09 dovecot: config: Warning: service auth {
client_limit=16384} is lower than required under max. load (528384)
> ---
> 
> 1. Where's the difference in Dovecot's logic between a mail service
that
> has a service count of 1 versus one with >1?
With service_count=1 it disconnects from auth immediately after logging in. With
service_count>0 the auth connection is kept open for the entire existence of
the imap process. This is mainly because after dropping privileges it
wouldn't be able to connect to the auth-master socket again. In theory if
the socket permissions were changed, it could keep reconnecting to auth-master
and not keep connections open all the time.
> 2. Any way to get the process recycling for IMAP going w/o setting the fd
> limit to a ridiculous amount? 

How about shrinking the imap process_limit? I highly doubt you can actually run
500k imap processes per server and have it still working. The largest I've
ever heard people running has been 50k processes per server.

On Tue, 29 Nov 2016 14:30:37 +0200 Timo Sirainen wrote:
> On 29 Nov 2016, at 2.57, Christian Balzer <chibi at gol.com> wrote:
> > 
> > service imap {
> >  # Most of the memory goes to mmap()ing files. You may need to
increase this
> >  # limit if you have huge mailboxes.
> >  #vsz_limit = $default_vsz_limit
> >  vsz_limit = 512M
> > 
> >  # Max. number of IMAP processes (connections)
> >  #process_limit = 1024
> >  process_limit = 524288
> > }
> ..
> > But adding a "service_count = 100" line (any value larger
than 1 really) to
> > the imap section we get the dreaded: 
> > ---
> > Nov 28 17:05:40 mbx09 dovecot: config: Warning: service auth {
client_limit=16384} is lower than required under max. load (528384)
> > ---
> > 
> > 1. Where's the difference in Dovecot's logic between a mail
service that
> > has a service count of 1 versus one with >1?
> 
> With service_count=1 it disconnects from auth immediately after logging in.
With service_count>0 the auth connection is kept open for the entire
existence of the imap process. This is mainly because after dropping privileges
it wouldn't be able to connect to the auth-master socket again. In theory if
the socket permissions were changed, it could keep reconnecting to auth-master
and not keep connections open all the time.
> 
Alright then, that's what I was suspecting. 
Too bad, but totally understandable. 
> > 2. Any way to get the process recycling for IMAP going w/o setting the
fd
> > limit to a ridiculous amount? 
> 
> 
> How about shrinking the imap process_limit? I highly doubt you can actually
run 500k imap processes per server and have it still working. The largest
I've ever heard people running has been 50k processes per server.
> 
Well, that's the limit these servers could theoretically take IOPS wise,
other things like memory might curtail that earlier. 
Also this is the fail-over level of this cluster pair, a single node
normally would only have to handle half of this.

Incidentally these 2 servers are currently running about 45k IMAP
processes each and the most busy process is (unsurprisingly) dovecot, the
master. 
But that's only using about 20% of one core and the system is currently
operating with the on-demand CPU governor, so that core is only at half
speed typically.
It's a pure SSD system, I/O utilization tends to peak around 3% and
averages less than 1%.
Memory is still half "free" (page cache) and an upgrade of that is
planned.

So from where I'm standing 100k per server (200k in fail-over) at the least
should be achievable easily.

Guess cranking up the fd limit it is then, still got 10 million spares
after all.

Thanks for the feedback,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/