> 1. 256GB of real RAM, swap is for chums.Are you sure that 100,000 IMAP sessions wouldn?t work well with SWAP, especially with fast SSD storage (which is a lot cheaper than RAM)? Seems that these IMAP processes are long lived processes (idling most of the time) that don?t need that much of the contents of real memory available for much of the life of the process. I use a database proxy in front of MySQL (for my web apps) so that there can be a large number of TCP connections to the proxy where the frontend requests are queued for execution using a small number of backend connections. Could Dovecot IMAP be re-written to be more efficient so it works more like MySQL (or other scalable data servers) that could handle a million or more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions aren?t doing much most of the time and shouldn?t really average 2MB of active data per session that needs to be resident in main memory at all times. My mail server isn?t that large yet as I haven?t fully deployed Dovecot outside my own small group yet, but it would be nice if scaling Dovecot IMAP to millions of users wasn?t limited to 50,000 IMAP sessions on a server...> On Feb 10, 2017, at 11:07 AM, Christian Balzer <chibi at gol.com> wrote: > > On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > >>> 1500 IMAP sessions will eat up about 3GB alone. >> >> Are you saying that Dovecot needs 2MB of physical memory per IMAP session? >> > That depends on the IMAP session, read the mailbox size and index size, > etc. > Some are significantly larger: > --- > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap > --- > > But yes, as somebody who has mailbox servers with 55k+ session the average > is around 1.6MB. > >> If I want to support a max 100,000 IMAP sessions per server, I should configure the server to have at least 200GBs of SWAP? >> > You will want:> 2. Understanding how to tune Dovecot and more importantly the overall > system to such a task (see that PID up there?). > 3. Be willing to deal with stuff like top and ps taking ages to start/run > and others like atop actually killing dovecot (performance wise, not > literally) when doing their obviously flawed cleanup on exit. Some things > clearly do NOT scale well. > > My current goal is to have 100k capable servers that work well, 200k in a > failover scenario, but that won't be particular enjoyable. > > Christian > >>> On Feb 10, 2017, at 3:58 AM, Christian Balzer <chibi at gol.com> wrote: >>> >>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: >>> >>>> hello >>>> >>>> could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. >>>> >>> >>> Be very precise here, you expect to see 1500 as the result of >>> "doveadm who |grep pop3 |wc -l"? >>> >>> Because that implies an ungodly number of POP3 connects per second, given >>> the typically short duration of these. >>> >>> 1500 IMAP connections (note that frequently a client will have more than >>> the INBOX open and thus have more than one session and thus process on the >>> server) are a much easier proposition, provided they are of the typical >>> long lasting type. >>> >>> So can you put a number to your expected logins per second (both protocols)? >>> >>>> my server >>>> >>>> server configuration >>>> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 >>>> gb hdd for data (No raid) >>>> >>> No RAID and no other replication like DRBD? >>> Why would you even bother? >>> >>> How many users/mailboxes in total with what quota? >>> >>> 1500 IMAP sessions will eat up about 3GB alone. >>> You will want more memory, simply to keep all relevant SLAB bits (inodes, >>> dentries) in RAM. >>> >>> If you really have several hundreds logins/s, you're facing several >>> bottlenecks: >>> 1. Login processes themselves (easily fixed by high performance mode) >>> 2. Auth processes (that will depend on your backends, method mostly) >>> 3. Dovecot master process (spawning mail processes) >>> >>> The later is a single-threaded process, so it will benefit from a faster >>> CPU core. >>> It can be dramatically improved by enabling process re-usage, see: >>> http://wiki.dovecot.org/PerformanceTuning >>> >>> However that also means more memory usage. >>> >>> >>> >>> Christian >>> >>>> >>>> thanks >>>> rajesh >>>> >>> >>> [snip] >>> -- >>> Christian Balzer Network/Systems Engineer >>> chibi at gol.com Global OnLine Japan/Rakuten Communications >>> http://www.gol.com/ >> > > > -- > Christian Balzer Network/Systems Engineer > chibi at gol.com <mailto:chibi at gol.com> Global OnLine Japan/Rakuten Communications > http://www.gol.com/ <http://www.gol.com/>
Hello, On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote:> > 1. 256GB of real RAM, swap is for chums. > > Are you sure that 100,000 IMAP sessions wouldn?t work well with SWAP, especially with fast SSD storage (which is a lot cheaper than RAM)? >I'm sure about tax and death, not much else. But as a rule of thumb I'd avoid swapping out stuff on production servers, even if it were to SSDs. Incidentally the servers I'm talking about here have their OS and swap on Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC S3610s. Relying on the kernel to make swap decisions is likely to result in much reduced performance even with fast SWAP when you're overcommitting things on that scale. But read on.> Seems that these IMAP processes are long lived processes (idling most of the time) that don?t need that much of the contents of real memory available for much of the life of the process. I use a database proxy in front of MySQL (for my web apps) so that there can be a large number of TCP connections to the proxy where the frontend requests are queued for execution using a small number of backend connections. > > Could Dovecot IMAP be re-written to be more efficient so it works more like MySQL (or other scalable data servers) that could handle a million or more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions aren?t doing much most of the time and shouldn?t really average 2MB of active data per session that needs to be resident in main memory at all times. >See IMAP hibernation: https://www.mail-archive.com/dovecot at dovecot.org/msg63429.html I'm going to deploy/test this in production in about 2 months from now, but if you look at the link and the consequent changelog entries you'll see that it has certain shortcomings and bug fixes in pretty much each release after it was introduced. But this is the correct way to tackle things, not SWAP. Alas I'm not expecting miracles and if more than 20% of the IMAP sessions here will be hibernated at any given time I'd be pleasantly surprised. Because between: 1. Finding a sensible imap_hibernate_timeout. 2. Having well behaved clients that keep idling instead of restarting the sequence (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/) 3. Having lots of mobile clients who either get disconnected (invisible to Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts (a large mobile carrier here times out idle TCP sessions after 2 minutes, forcing people to use 1 minute IDLE renewals, making 1. up there a nightmare). 4. Having really broken clients (don't ask, I can't tell) which open IMAP sessions, don't put them into IDLE and thus having them expire after 30 minutes. the pool of eligible IDLE sessions isn't as big as it could be, in my case at least.> My mail server isn?t that large yet as I haven?t fully deployed Dovecot outside my own small group yet, but it would be nice if scaling Dovecot IMAP to millions of users wasn?t limited to 50,000 IMAP sessions on a server... >Scaling up is nice and desirable from a cost (rack space, HW) perspective, but the scalability of things OTHER than Dovecot as I pointed out plus that little detail of failure domains (do you really want half of your eggs in one basket?) argue for scaling out after a certain density. I'm feeling my way there at this time, but expect more than 100k sessions per server to be tricky. Lastly, when I asked about 500k sessions per server here not so long ago, ( http://www.dovecot.org/list/dovecot/2016-November/106284.html ) Timo mentioned that he's not aware of anybody doing more than 50k per server, something I got licked already and definitely will go to 100k eventually. Regards, Christian> > On Feb 10, 2017, at 11:07 AM, Christian Balzer <chibi at gol.com> wrote: > > > > On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > > > >>> 1500 IMAP sessions will eat up about 3GB alone. > >> > >> Are you saying that Dovecot needs 2MB of physical memory per IMAP session? > >> > > That depends on the IMAP session, read the mailbox size and index size, > > etc. > > Some are significantly larger: > > --- > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap > > --- > > > > But yes, as somebody who has mailbox servers with 55k+ session the average > > is around 1.6MB. > > > >> If I want to support a max 100,000 IMAP sessions per server, I should configure the server to have at least 200GBs of SWAP? > >> > > You will want: > > > 2. Understanding how to tune Dovecot and more importantly the overall > > system to such a task (see that PID up there?). > > 3. Be willing to deal with stuff like top and ps taking ages to start/run > > and others like atop actually killing dovecot (performance wise, not > > literally) when doing their obviously flawed cleanup on exit. Some things > > clearly do NOT scale well. > > > > My current goal is to have 100k capable servers that work well, 200k in a > > failover scenario, but that won't be particular enjoyable. > > > > Christian > > > >>> On Feb 10, 2017, at 3:58 AM, Christian Balzer <chibi at gol.com> wrote: > >>> > >>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > >>> > >>>> hello > >>>> > >>>> could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. > >>>> > >>> > >>> Be very precise here, you expect to see 1500 as the result of > >>> "doveadm who |grep pop3 |wc -l"? > >>> > >>> Because that implies an ungodly number of POP3 connects per second, given > >>> the typically short duration of these. > >>> > >>> 1500 IMAP connections (note that frequently a client will have more than > >>> the INBOX open and thus have more than one session and thus process on the > >>> server) are a much easier proposition, provided they are of the typical > >>> long lasting type. > >>> > >>> So can you put a number to your expected logins per second (both protocols)? > >>> > >>>> my server > >>>> > >>>> server configuration > >>>> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 > >>>> gb hdd for data (No raid) > >>>> > >>> No RAID and no other replication like DRBD? > >>> Why would you even bother? > >>> > >>> How many users/mailboxes in total with what quota? > >>> > >>> 1500 IMAP sessions will eat up about 3GB alone. > >>> You will want more memory, simply to keep all relevant SLAB bits (inodes, > >>> dentries) in RAM. > >>> > >>> If you really have several hundreds logins/s, you're facing several > >>> bottlenecks: > >>> 1. Login processes themselves (easily fixed by high performance mode) > >>> 2. Auth processes (that will depend on your backends, method mostly) > >>> 3. Dovecot master process (spawning mail processes) > >>> > >>> The later is a single-threaded process, so it will benefit from a faster > >>> CPU core. > >>> It can be dramatically improved by enabling process re-usage, see: > >>> http://wiki.dovecot.org/PerformanceTuning > >>> > >>> However that also means more memory usage. > >>> > >>> > >>> > >>> Christian > >>> > >>>> > >>>> thanks > >>>> rajesh > >>>> > >>> > >>> [snip] > >>> -- > >>> Christian Balzer Network/Systems Engineer > >>> chibi at gol.com Global OnLine Japan/Rakuten Communications > >>> http://www.gol.com/ > >> > > > > > > -- > > Christian Balzer Network/Systems Engineer > > chibi at gol.com <mailto:chibi at gol.com> Global OnLine Japan/Rakuten Communications > > http://www.gol.com/ <http://www.gol.com/>-- Christian Balzer Network/Systems Engineer chibi at gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Thanks for the info. I do have one further question for you. On your servers that are currently handling 50k IMAP sessions, how many users does that correspond to? Since many users will have multiple IMAP sessions on multiple devices, I?d like to hear about some real-world numbers that could be used for budgeting a new project like mine. Also, do you use Dovecot IMAP proxies in front of your backend servers? If so, how many IMAP sessions can one proxy server handle (assuming the proxy does authorization using MySQL running on a separate server)? And, could the proxy server be tuned to help in optimizing mostly IDLE backend sessions?> On Feb 12, 2017, at 1:58 AM, Christian Balzer <chibi at gol.com> wrote: > > > Hello, > > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote: > >>> 1. 256GB of real RAM, swap is for chums. >> >> Are you sure that 100,000 IMAP sessions wouldn?t work well with SWAP, especially with fast SSD storage (which is a lot cheaper than RAM)? >> > > I'm sure about tax and death, not much else. > > But as a rule of thumb I'd avoid swapping out stuff on production servers, > even if it were to SSDs. > Incidentally the servers I'm talking about here have their OS and swap on > Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC > S3610s. > > Relying on the kernel to make swap decisions is likely to result in much > reduced performance even with fast SWAP when you're overcommitting things > on that scale. > > > But read on. > >> Seems that these IMAP processes are long lived processes (idling most of the time) that don?t need that much of the contents of real memory available for much of the life of the process. I use a database proxy in front of MySQL (for my web apps) so that there can be a large number of TCP connections to the proxy where the frontend requests are queued for execution using a small number of backend connections. >> >> Could Dovecot IMAP be re-written to be more efficient so it works more like MySQL (or other scalable data servers) that could handle a million or more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions aren?t doing much most of the time and shouldn?t really average 2MB of active data per session that needs to be resident in main memory at all times. >> > See IMAP hibernation: > https://www.mail-archive.com/dovecot at dovecot.org/msg63429.html <https://www.mail-archive.com/dovecot at dovecot.org/msg63429.html> > > I'm going to deploy/test this in production in about 2 months from now, > but if you look at the link and the consequent changelog entries you'll see > that it has certain shortcomings and bug fixes in pretty much each release > after it was introduced. > > But this is the correct way to tackle things, not SWAP. > > Alas I'm not expecting miracles and if more than 20% of the IMAP sessions > here will be hibernated at any given time I'd be pleasantly surprised. > > Because between: > > 1. Finding a sensible imap_hibernate_timeout. > > 2. Having well behaved clients that keep idling instead of restarting the > sequence (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/ <https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/>) > > 3. Having lots of mobile clients who either get disconnected (invisible to > Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts > (a large mobile carrier here times out idle TCP sessions after 2 minutes, > forcing people to use 1 minute IDLE renewals, making 1. up there a > nightmare). > > 4. Having really broken clients (don't ask, I can't tell) which open IMAP > sessions, don't put them into IDLE and thus having them expire after 30 > minutes. > > the pool of eligible IDLE sessions isn't as big as it could be, in my case > at least. > >> My mail server isn?t that large yet as I haven?t fully deployed Dovecot outside my own small group yet, but it would be nice if scaling Dovecot IMAP to millions of users wasn?t limited to 50,000 IMAP sessions on a server... >> > > Scaling up is nice and desirable from a cost (rack space, HW) perspective, > but the scalability of things OTHER than Dovecot as I pointed out plus > that little detail of failure domains (do you really want half of your > eggs in one basket?) argue for scaling out after a certain density. > > I'm feeling my way there at this time, but expect more than 100k sessions > per server to be tricky. > > Lastly, when I asked about 500k sessions per server here not so long ago, > ( http://www.dovecot.org/list/dovecot/2016-November/106284.html <http://www.dovecot.org/list/dovecot/2016-November/106284.html> ) > Timo mentioned that he's not aware of anybody doing more than 50k per > server, something I got licked already and definitely will go to 100k > eventually. > > Regards, > > Christian >>> On Feb 10, 2017, at 11:07 AM, Christian Balzer <chibi at gol.com> wrote: >>> >>> On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: >>> >>>>> 1500 IMAP sessions will eat up about 3GB alone. >>>> >>>> Are you saying that Dovecot needs 2MB of physical memory per IMAP session? >>>> >>> That depends on the IMAP session, read the mailbox size and index size, >>> etc. >>> Some are significantly larger: >>> --- >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap >>> --- >>> >>> But yes, as somebody who has mailbox servers with 55k+ session the average >>> is around 1.6MB. >>> >>>> If I want to support a max 100,000 IMAP sessions per server, I should configure the server to have at least 200GBs of SWAP? >>>> >>> You will want: >> >>> 2. Understanding how to tune Dovecot and more importantly the overall >>> system to such a task (see that PID up there?). >>> 3. Be willing to deal with stuff like top and ps taking ages to start/run >>> and others like atop actually killing dovecot (performance wise, not >>> literally) when doing their obviously flawed cleanup on exit. Some things >>> clearly do NOT scale well. >>> >>> My current goal is to have 100k capable servers that work well, 200k in a >>> failover scenario, but that won't be particular enjoyable. >>> >>> Christian >>> >>>>> On Feb 10, 2017, at 3:58 AM, Christian Balzer <chibi at gol.com> wrote: >>>>> >>>>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: >>>>> >>>>>> hello >>>>>> >>>>>> could somebody with experience let me know the dovecot config file settings to handle around 1500 simultaneous connections over pop3 and 1500 connection over imap simultaneously. >>>>>> >>>>> >>>>> Be very precise here, you expect to see 1500 as the result of >>>>> "doveadm who |grep pop3 |wc -l"? >>>>> >>>>> Because that implies an ungodly number of POP3 connects per second, given >>>>> the typically short duration of these. >>>>> >>>>> 1500 IMAP connections (note that frequently a client will have more than >>>>> the INBOX open and thus have more than one session and thus process on the >>>>> server) are a much easier proposition, provided they are of the typical >>>>> long lasting type. >>>>> >>>>> So can you put a number to your expected logins per second (both protocols)? >>>>> >>>>>> my server >>>>>> >>>>>> server configuration >>>>>> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 >>>>>> gb hdd for data (No raid) >>>>>> >>>>> No RAID and no other replication like DRBD? >>>>> Why would you even bother? >>>>> >>>>> How many users/mailboxes in total with what quota? >>>>> >>>>> 1500 IMAP sessions will eat up about 3GB alone. >>>>> You will want more memory, simply to keep all relevant SLAB bits (inodes, >>>>> dentries) in RAM. >>>>> >>>>> If you really have several hundreds logins/s, you're facing several >>>>> bottlenecks: >>>>> 1. Login processes themselves (easily fixed by high performance mode) >>>>> 2. Auth processes (that will depend on your backends, method mostly) >>>>> 3. Dovecot master process (spawning mail processes) >>>>> >>>>> The later is a single-threaded process, so it will benefit from a faster >>>>> CPU core. >>>>> It can be dramatically improved by enabling process re-usage, see: >>>>> http://wiki.dovecot.org/PerformanceTuning >>>>> >>>>> However that also means more memory usage. >>>>> >>>>> >>>>> >>>>> Christian >>>>> >>>>>> >>>>>> thanks >>>>>> rajesh >>>>>> >>>>> >>>>> [snip] >>>>> -- >>>>> Christian Balzer Network/Systems Engineer >>>>> chibi at gol.com Global OnLine Japan/Rakuten Communications >>>>> http://www.gol.com/ >>>> >>> >>> >>> -- >>> Christian Balzer Network/Systems Engineer >>> chibi at gol.com <mailto:chibi at gol.com> <mailto:chibi at gol.com <mailto:chibi at gol.com>> Global OnLine Japan/Rakuten Communications >>> http://www.gol.com/ <http://www.gol.com/> <http://www.gol.com/ <http://www.gol.com/>> > > > -- > Christian Balzer Network/Systems Engineer > chibi at gol.com <mailto:chibi at gol.com> Global OnLine Japan/Rakuten Communications > http://www.gol.com/ <http://www.gol.com/>