Hello,
On Wed, 5 Apr 2017 23:45:33 -0700 Mark Moseley wrote:
> We've been using hibernate for about half a year with no ill effects.
There
> were various logged errors in earlier versions of dovecot, but even with
> those, we never heard a reported customer-side error (almost always when
> transitioning from hibernate back to regular imap; in the case of those
> errors, presumably the mail client just reconnected silently).
>
This is my impression as well (silent reconnect w/o anything really bad
happening) with regard to the bug I just reported/saw.
> For no particular reason besides wanting to start conservatively, we've
got
> client_limit set to 50 on the hibernate procs (with 1100 total hibernated
> connections on the box I'm looking at). At only a little over a meg
each,
> I'm fine with those extra processes.
>
Yeah, but 50 would be a tad too conservative for our purposes here.
I'll keep an eye on it and see how it goes, first checkpoint would be at
1k hibernated sessions. ^_^
Christian
> On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi <aki.tuomi at dovecot.fi>
wrote:
>
> >
> >
> > On 06.04.2017 06:15, Christian Balzer wrote:
> > > Hello,
> > >
> > > as some may remember, we're running very dense IMAP cluster
here, in
> > > excess of 50k IMAP sessions per node (current record holder is
68k,
> > design
> > > is for 200k+).
> > >
> > > The first issue we ran into was that the dovecot master process
(which is
> > > single thread and thus a bottleneck) was approaching 100% CPU
usage (aka
> > > using a full core) when trying to spawn off new IMAP processes.
> > >
> > > This was rectified by giving IMAP a service count of 200 to
create a pool
> > > of "idling" processes eventually, reducing the strain
for the master
> > > process dramatically. That of course required generous cranking
up
> > > ulimits, FDs in particular.
> > >
> > > The next issues of course is (as mentioned before) the memory
usage of
> > all
> > > those IMAP processes and the fact that quite a few things outside
of
> > > dovecote (ps, etc) tend to get quite sedate when dealing with
tens of
> > > thousands of processes.
> > >
> > > We just started to deploy a new mailbox cluster pair with 2.2.27
and
> > > having IMAP hibernate configured.
> > > Getting this work is a PITA though with regards to ownership and
access
> > > rights to the various sockets, this part could definitely do with
some
> > > better (I know, difficult) defaults or at least more
documentation (none
> > > besides the source and this ML).
> > >
> > > Initial results are very promising, depending on what your
clients are
> > > doing (are they well behaved, are your users constantly looking
at other
> > > folders, etc) the vast majority of IDLE processes will be in
hibernated
> > > at any given time and thus not only using a fraction of the RAM
otherwise
> > > needed but also freeing up process space.
> > >
> > > Real life example:
> > > 240 users, 86 imap processes (80% of those not IDLE) and:
> > > dovecot 119157 0.0 0.0 10452 3236 ? S Apr01
0:21
> > dovecot/imap-hibernate [237 connections]
> > > That's 237 hibernated connections and thus less processes
than otherwise.
> > >
> > >
> > > I assume that given the silence on the ML what we are going to be
the
> > > first hibernate users where the term "large scale" does
apply.
> > > Despite that I have some questions, clarifications/confirmations:
> > >
> > > Our current default_client_limit is 16k, which can be seen by
having 5
> > > config processes on our 65k+ session node. ^_-
> > >
> > > This would also apply to imap-hibernate, one wonders if
that's fine
> > > (config certainly has no issues) or if something smaller would be
> > > appropriate here?
> > >
> > > Since we have idling IMAP processes around most of the time, the
strain
> > of
> > > re-spawning proper processes from imap-hibernate should be just
as
> > reduced
> > > as for the dovecot master, correct?
> > >
> > >
> > > I'll keep reporting our experiences here, that is if
something blows up
> > > spectacularly. ^o^
> > >
> > > Christian
> >
> > Hi!
> >
> > We have customers using it in larger deployments. A good idea is to
have
> > as much of your clients hibernating as possible, as the hibernation
> > process is much smaller than actual IMAP process.
> >
> > You should probably also look at reusing the processes, as this will
> > probably help your performance,
> > https://wiki.dovecot.org/PerformanceTuning and
> > https://wiki.dovecot.org/LoginProcess are probably a good starting
> > point, although I suspect you've read these already.
> >
> > If you are running a dense server, cranking up various limits is
rather
> > expected.
> >
> > Aki
> >
>
--
Christian Balzer Network/Systems Engineer
chibi at gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/