Hello, as some may remember, we're running very dense IMAP cluster here, in excess of 50k IMAP sessions per node (current record holder is 68k, design is for 200k+). The first issue we ran into was that the dovecot master process (which is single thread and thus a bottleneck) was approaching 100% CPU usage (aka using a full core) when trying to spawn off new IMAP processes. This was rectified by giving IMAP a service count of 200 to create a pool of "idling" processes eventually, reducing the strain for the master process dramatically. That of course required generous cranking up ulimits, FDs in particular. The next issues of course is (as mentioned before) the memory usage of all those IMAP processes and the fact that quite a few things outside of dovecote (ps, etc) tend to get quite sedate when dealing with tens of thousands of processes. We just started to deploy a new mailbox cluster pair with 2.2.27 and having IMAP hibernate configured. Getting this work is a PITA though with regards to ownership and access rights to the various sockets, this part could definitely do with some better (I know, difficult) defaults or at least more documentation (none besides the source and this ML). Initial results are very promising, depending on what your clients are doing (are they well behaved, are your users constantly looking at other folders, etc) the vast majority of IDLE processes will be in hibernated at any given time and thus not only using a fraction of the RAM otherwise needed but also freeing up process space. Real life example: 240 users, 86 imap processes (80% of those not IDLE) and: dovecot 119157 0.0 0.0 10452 3236 ? S Apr01 0:21 dovecot/imap-hibernate [237 connections] That's 237 hibernated connections and thus less processes than otherwise. I assume that given the silence on the ML what we are going to be the first hibernate users where the term "large scale" does apply. Despite that I have some questions, clarifications/confirmations: Our current default_client_limit is 16k, which can be seen by having 5 config processes on our 65k+ session node. ^_- This would also apply to imap-hibernate, one wonders if that's fine (config certainly has no issues) or if something smaller would be appropriate here? Since we have idling IMAP processes around most of the time, the strain of re-spawning proper processes from imap-hibernate should be just as reduced as for the dovecot master, correct? I'll keep reporting our experiences here, that is if something blows up spectacularly. ^o^ Christian -- Christian Balzer Network/Systems Engineer chibi at gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
On 06.04.2017 06:15, Christian Balzer wrote:> Hello, > > as some may remember, we're running very dense IMAP cluster here, in > excess of 50k IMAP sessions per node (current record holder is 68k, design > is for 200k+). > > The first issue we ran into was that the dovecot master process (which is > single thread and thus a bottleneck) was approaching 100% CPU usage (aka > using a full core) when trying to spawn off new IMAP processes. > > This was rectified by giving IMAP a service count of 200 to create a pool > of "idling" processes eventually, reducing the strain for the master > process dramatically. That of course required generous cranking up > ulimits, FDs in particular. > > The next issues of course is (as mentioned before) the memory usage of all > those IMAP processes and the fact that quite a few things outside of > dovecote (ps, etc) tend to get quite sedate when dealing with tens of > thousands of processes. > > We just started to deploy a new mailbox cluster pair with 2.2.27 and > having IMAP hibernate configured. > Getting this work is a PITA though with regards to ownership and access > rights to the various sockets, this part could definitely do with some > better (I know, difficult) defaults or at least more documentation (none > besides the source and this ML). > > Initial results are very promising, depending on what your clients are > doing (are they well behaved, are your users constantly looking at other > folders, etc) the vast majority of IDLE processes will be in hibernated > at any given time and thus not only using a fraction of the RAM otherwise > needed but also freeing up process space. > > Real life example: > 240 users, 86 imap processes (80% of those not IDLE) and: > dovecot 119157 0.0 0.0 10452 3236 ? S Apr01 0:21 dovecot/imap-hibernate [237 connections] > That's 237 hibernated connections and thus less processes than otherwise. > > > I assume that given the silence on the ML what we are going to be the > first hibernate users where the term "large scale" does apply. > Despite that I have some questions, clarifications/confirmations: > > Our current default_client_limit is 16k, which can be seen by having 5 > config processes on our 65k+ session node. ^_- > > This would also apply to imap-hibernate, one wonders if that's fine > (config certainly has no issues) or if something smaller would be > appropriate here? > > Since we have idling IMAP processes around most of the time, the strain of > re-spawning proper processes from imap-hibernate should be just as reduced > as for the dovecot master, correct? > > > I'll keep reporting our experiences here, that is if something blows up > spectacularly. ^o^ > > ChristianHi! We have customers using it in larger deployments. A good idea is to have as much of your clients hibernating as possible, as the hibernation process is much smaller than actual IMAP process. You should probably also look at reusing the processes, as this will probably help your performance, https://wiki.dovecot.org/PerformanceTuning and https://wiki.dovecot.org/LoginProcess are probably a good starting point, although I suspect you've read these already. If you are running a dense server, cranking up various limits is rather expected. Aki
We've been using hibernate for about half a year with no ill effects. There were various logged errors in earlier versions of dovecot, but even with those, we never heard a reported customer-side error (almost always when transitioning from hibernate back to regular imap; in the case of those errors, presumably the mail client just reconnected silently). For no particular reason besides wanting to start conservatively, we've got client_limit set to 50 on the hibernate procs (with 1100 total hibernated connections on the box I'm looking at). At only a little over a meg each, I'm fine with those extra processes. On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:> > > On 06.04.2017 06:15, Christian Balzer wrote: > > Hello, > > > > as some may remember, we're running very dense IMAP cluster here, in > > excess of 50k IMAP sessions per node (current record holder is 68k, > design > > is for 200k+). > > > > The first issue we ran into was that the dovecot master process (which is > > single thread and thus a bottleneck) was approaching 100% CPU usage (aka > > using a full core) when trying to spawn off new IMAP processes. > > > > This was rectified by giving IMAP a service count of 200 to create a pool > > of "idling" processes eventually, reducing the strain for the master > > process dramatically. That of course required generous cranking up > > ulimits, FDs in particular. > > > > The next issues of course is (as mentioned before) the memory usage of > all > > those IMAP processes and the fact that quite a few things outside of > > dovecote (ps, etc) tend to get quite sedate when dealing with tens of > > thousands of processes. > > > > We just started to deploy a new mailbox cluster pair with 2.2.27 and > > having IMAP hibernate configured. > > Getting this work is a PITA though with regards to ownership and access > > rights to the various sockets, this part could definitely do with some > > better (I know, difficult) defaults or at least more documentation (none > > besides the source and this ML). > > > > Initial results are very promising, depending on what your clients are > > doing (are they well behaved, are your users constantly looking at other > > folders, etc) the vast majority of IDLE processes will be in hibernated > > at any given time and thus not only using a fraction of the RAM otherwise > > needed but also freeing up process space. > > > > Real life example: > > 240 users, 86 imap processes (80% of those not IDLE) and: > > dovecot 119157 0.0 0.0 10452 3236 ? S Apr01 0:21 > dovecot/imap-hibernate [237 connections] > > That's 237 hibernated connections and thus less processes than otherwise. > > > > > > I assume that given the silence on the ML what we are going to be the > > first hibernate users where the term "large scale" does apply. > > Despite that I have some questions, clarifications/confirmations: > > > > Our current default_client_limit is 16k, which can be seen by having 5 > > config processes on our 65k+ session node. ^_- > > > > This would also apply to imap-hibernate, one wonders if that's fine > > (config certainly has no issues) or if something smaller would be > > appropriate here? > > > > Since we have idling IMAP processes around most of the time, the strain > of > > re-spawning proper processes from imap-hibernate should be just as > reduced > > as for the dovecot master, correct? > > > > > > I'll keep reporting our experiences here, that is if something blows up > > spectacularly. ^o^ > > > > Christian > > Hi! > > We have customers using it in larger deployments. A good idea is to have > as much of your clients hibernating as possible, as the hibernation > process is much smaller than actual IMAP process. > > You should probably also look at reusing the processes, as this will > probably help your performance, > https://wiki.dovecot.org/PerformanceTuning and > https://wiki.dovecot.org/LoginProcess are probably a good starting > point, although I suspect you've read these already. > > If you are running a dense server, cranking up various limits is rather > expected. > > Aki >