dovecot - Jul 2015 - Scalability with high density servers and proxies, TCP port limits

Hello,

first post in 3 years, kinda shows how painless Dovecot is. ^o^

Also this isn't really a dovecot issue, alas it's involved and since
there
are some large scale implementations of it I hope somebody here has some
insights I might have missed.

Currently we're running this setup:

1. LVS (DR mode) in a HA configuration (2 node cluster)
2. Dovecot in proxy mode on a 2 node cluster
3. Dovecot on actual mailbox servers (dual node DRBD clusters)

There are about 500k users, but most of them use POP3, so there are
usually less than 6k IMAP sesions at any given time.

This is about to change, I'm looking at potentially millions of users who
will have all semi-permanent IMAP sessions.

We already have a pure SSD based mailbox cluster and based on the
experiences with that another one is on order that will be able to easily
handle about 500k users with regards to IOPS and other needs.

However there's the issue of having all these concurrent IMAP sessions.
Namely, running out of ephemeral ports.

Lets assume 2 million users and 50k ports per IP and revisit the setup
above.

1. LVS should have no problem, from experience and tests I expect a well
tuned and spec'ed machine to handle millions of connections.
This is in DR mode, in NAT mode I assume things would run into a wall a
lot quicker.
But even if LVS should run out of steam, there's a wide selection of high
capacity load balancers available.

2. Here is where the fun starts. 
Each IMAP session that gets proxied to the real mailbox server needs a
port for the outgoing connection. 
So to support 2 million sessions we need 40 IP addresses here. Ouch.
And from a brief test having multiple IP addresses per server won't help
either (Dovecot unsurprisingly picks the main IP when establishing a
proxy session to the real mailbox), at least not with just one default GW. 

3. All of this gets repeated on the actual mailbox servers, by either
having a lot of low density servers or (preferably) high density servers
with multiple IP addresses. 

Am I on track so far or missing something obvious?

How many concurrent connections do you (hello Timo) think dovecot in proxy
mode can handle? High performance mode of course in this case.
I'm interested in internal limitations, assume that CPU and RAM are
amply supplied.

Any and all feedback is appreciated.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/

Hi,

Am 03.07.2015 um 05:14 schrieb Christian Balzer:
>
> 2. Here is where the fun starts.
> Each IMAP session that gets proxied to the real mailbox server needs a
> port for the outgoing connection.
> So to support 2 million sessions we need 40 IP addresses here. Ouch.
> And from a brief test having multiple IP addresses per server won't
help
> either (Dovecot unsurprisingly picks the main IP when establishing a
> proxy session to the real mailbox), at least not with just one default GW.
>
If I remeber correctly there is a config option in dovecot 2.x where you 
can set the ip addresses which dovecot should use for outgoing proxy 
connections. Sorry, but I can't remeber the option.

Best
Urban

On Fri, 03 Jul 2015 07:05:43 +0200 Urban Loesch wrote:
> Hi,
> 
> Am 03.07.2015 um 05:14 schrieb Christian Balzer:
> 
> >
> > 2. Here is where the fun starts.
> > Each IMAP session that gets proxied to the real mailbox server needs a
> > port for the outgoing connection.
> > So to support 2 million sessions we need 40 IP addresses here. Ouch.
> > And from a brief test having multiple IP addresses per server
won't
> > help either (Dovecot unsurprisingly picks the main IP when
> > establishing a proxy session to the real mailbox), at least not with
> > just one default GW.
> >
> 
> If I remeber correctly there is a config option in dovecot 2.x where you 
> can set the ip addresses which dovecot should use for outgoing proxy 
> connections. Sorry, but I can't remeber the option.
> 
Looking at the documentation on the Wiki I was going to say "That won't
help", as it says address.
http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy

But since that page is rather terse, I looked up the changelog and found
that it indeed was added for use cases like mine:
http://www.dovecot.org/list/dovecot-cvs/2014-June/024574.html

Unfortunately the latest dovecot version in Debian is 2.2.13...

Additionally this still leaves the actual mailbox servers, which in my
case will need to be able to handle more than 50k sessions as well. 

Thanks for the info,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/