On Wed, 15 Feb 2017 08:43:23 +0530 Rajesh M wrote:
> christian
>
> the servers i currently own are dell servers. The servers i plan to buy are
Dell R530, 2U rack servers with 8 x 3.5 inch drives, with 64 gb ram each,
Hardware raid. I am thinking of 2 X 300 gb ssds raid1 and 6 x 2 tb drives in
raid10 for data. I do not have any experience in setting up drdb (that would be
my next step) ... primarily using standalone servers with hardware level
redundancy. For backup-ups i rsync data to backup servers, also use qmail taps
for backing up sent/recd emails on a per user basis.
>
> i am using mysql authentication backend
> my mail server is qmailtoaster, ie with vpopmail, spamdyke, simscan.
>
> you had mentioned as follows
> ##############
> With 4000 users and 5 logins/s I can't imagine anything that
wouldn't be
> able to:
> a) put all of the relevant data into RAM and
> b) thus handle auth requests in very timely manner.
> The example I gave earlier about my servers with 140 logins/s
> authenticates against LDAP and isn't very busy doing so while also
> fitting all 2.7GB of data for about 1 million users into RAM.
> ##############
>
> >>>>>>>>>>>>>>>>>>how
do i put all relevant data into the RAM ?
>
By configuring MySQL accordingly of course, for example:
--- my.cnf:
# Set buffer pool size to 50-80% of your computer's memory,
# but make sure on Linux x86 total memory usage is < 2GB
innodb_buffer_pool_size=8G
innodb_additional_mem_pool_size=32M
---
--- top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
39045 mysql 20 0 9170m 784m 10m S 0 2.4 3:38.69 mysqld
----
--- tuning-primer:
INNODB STATUS
Current InnoDB index space = 49 M
Current InnoDB data space = 109 M
Current InnoDB buffer pool free = 98 %
Current innodb_buffer_pool_size = 8.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
---
Christian
> thanks
> rajesh
>
>
> ----- Original Message -----
> From: Christian Balzer [mailto:chibi at gol.com]
> To: dovecot at dovecot.org
> Sent: Tue, 14 Feb 2017 16:40:01 +0900
> Subject:
>
>
> Hello,
>
> On Mon, 13 Feb 2017 22:53:23 +0530 Rajesh M wrote:
>
> > thanks for your help
> >
> > happy to say that the performance dramatically improved after i use
the high performance settings from here
> > http://wiki.dovecot.org/LoginProcess
> >
> That's why that page is there.
>
> > grep Login: /var/log/mail.log.1 |wc -l
> > with the mail.log being of a typical, busy day.
> > 412992
> >
> So about 5 login/s, not that busy, but that of course also depends on your
> HW and authentication backend.
>
> > i also picked up the imap and pop3 connections during peak hours
> > [root at ns1 domains]# doveadm who | awk
'/imap/{m+=$2}/pop3/{n+=$2}END{print m,n}'
> > username # proto (pids)
(ips)
> >
> > max figures i got was 535 for imap and 97 for pop
> >
> >
> > Question 1
> > i wish to improve the performance further by caching the logins.
current the same is kept disable because when user's change passwords then
they are not able to immediately login with the new password for some time. How
to solve this issue.
> >
> There is no solution, caching auth data is by default fraught with that
> sort of peril.
>
> And how did you determine that caching auth data would actually improve
> things?
>
> What is your authentication backend?
>
> With 4000 users and 5 logins/s I can't imagine anything that
wouldn't be
> able to:
> a) put all of the relevant data into RAM and
> b) thus handle auth requests in very timely manner.
>
> The example I gave earlier about my servers with 140 logins/s
> authenticates against LDAP and isn't very busy doing so while also
> fitting all 2.7GB of data for about 1 million users into RAM.
>
> >
> >
> > further to the above.
> > i am planning to put up a large mail server with one of the following
options
> >
> Don't plan 1, always plan a pair and then use whatever you're
comfortable
> with to turn them into a HA cluster, like Pacemaker with DRBD.
>
> > option 1
> > OS and mailqueue : 2 X 600 gb 15k rpm raid 1
> What kind of RAID, HW or MD?
>
> > data : 4 X 2000 gb raid 10 for data
> > backup : 2 X 2000 gb raid 10 for backup
>
> While with Linux MD a 2 disk RAID10 would be possible, it wouldn't be
> particular helpful (slower than a RAID 1 with writes).
>
> > 32 gb ram
> >
> >
> > option 2
> > os and mail queue : 4 X 600 gb 15k rpm raid 10
> Those 15k HDDs tend to be expensive.
> Estimate your expected writes per day (iostat on an existing server can
> give you some hints) and get 2 SSDs instead, DC S3520s if 1 DWPD is
> sufficient, DC S3610s if you need 3 DWPD (doubtful), S3710 if you need 10
> DWPD (very doubtful).
> Similar for Samsung DC level SSDs, check their specs.
>
> For example, my busiest mailbox servers only write about 250KB/s averaged,
> which mean about 50GB/day and thus would be a fit even for a small S3520.
> I'm still using 200GB S3710s because I can afford them, like the speed
and
> never ever want to worry about wearout.
>
> > data : 4 X 2000 gb raid 10 for data
> > 32 gb ram
> >
> > i am having the OS and mail queue on primary drives
> >
> > i understand that Raid10 give a faster write and read whereas Raid 1
will have slow writes and fast reads
> >
> While that is true, you're missing the real difference here, the number
of
> HDDs. With 4 of them in the RAID10 they will be twice as fast as the 2
> disk RAID1.
> This can be tweaked even further with the various RAID 10 layout options,
> "man md".
>
> > the question is will there be huge performance difference between
Raid1 and Raid 10 when it comes small files like mail queue ?
> >
> Again as explained above, double the IOPS, so yes.
>
> Christian
> >
> > thanks
> > rajesh
> > ----- Original Message -----
> > From: Christian Balzer [mailto:chibi at gol.com]
> > To: dovecot at dovecot.org
> > Cc: kevin at my.walr.us
> > Sent: Mon, 13 Feb 2017 10:46:13 +0900
> > Subject:
> >
> >
> > Hello,
> >
> > On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote:
> >
> > > Thanks for the info. I do have one further question for you. On
your servers that are currently handling 50k IMAP sessions, how many users does
that correspond to? Since many users will have multiple IMAP sessions on
multiple devices, I?d like to hear about some real-world numbers that could be
used for budgeting a new project like mine.
> > >
> >
> > Those servers would actually be the wrong ones to look at, as they are
> > primarily accessed by the aforementioned broken client, so the numbers
are
> > somewhat skewed.
> > However looking at other servers with a more "normal" user
spread things
> > aren't actually too different.
> >
> > The average number of sessions per user tends to be 2.
> > The clear majority (over 50%) only has one session open (people with
well
> > behaved and configured clients watching the INBOX mostly).
> > Another 30% has 2 sessions open, again the typical state of this would
be
> > clients watching another mailbox besides INBOX, typically SENT.
> > The rest has 3 and more sessions open.
> >
> > The number of sessions could of course be drastically reduced if any
> > client would support IMAP NOTIFY, alas none that I'm aware of do.
> >
> > Lastly no more than 60% of the session seem to be in IDLE at any given
> > time, so my comments about RAM and IMAP hibernation effectiveness
stand.
> >
> > > Also, do you use Dovecot IMAP proxies in front of your backend
servers? If so, how many IMAP sessions can one proxy server handle (assuming the
proxy does authorization using MySQL running on a separate server)? And, could
the proxy server be tuned to help in optimizing mostly IDLE backend sessions?
> > >
> >
> > Yes to Dovecot Proxying, of course.
> >
> > No idea about MySQL, with (Open)LDAP nothing is breaking a sweat at an
> > average of 140 logins per second (IMAP and POP) on the 2 proxy
servers.
> > If you can fit your entire dataset into RAM it should be fine, my LDAP
> > servers fall into that category and take about 10% of a slow (1.2GHz,
34%,
> > power-save mode) core only to handle the that load (also 2 servers).
> > And the rate of logins/s is what you need to worry about most and
optimize
> > for.
> >
> > The proxies will of course have to do the shuffling of data and SSL
> > en/de-coding, but again they're not particular busy with that.
> >
> > The number of sessions comes into play when looking at the number of
login
> > processes on the proxies and their memory footprint.
> > An IMAP login process on the proxies in performance mode with a client
> > limit of 1000 will consume about 55MB at most.
> > So assume at least 55KB RAM per session.
> >
> > Read Timo's mail I linked to about IMAP hibernation, AFAIK nothing
has
> > happened to make proxies more supportive for this though.
> >
> > Christian
> > > > On Feb 12, 2017, at 1:58 AM, Christian Balzer <chibi at
gol.com> wrote:
> > > >
> > > >
> > > > Hello,
> > > >
> > > > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote:
> > > >
> > > >>> 1. 256GB of real RAM, swap is for chums.
> > > >>
> > > >> Are you sure that 100,000 IMAP sessions wouldn?t work
well with SWAP, especially with fast SSD storage (which is a lot cheaper than
RAM)?
> > > >>
> > > >
> > > > I'm sure about tax and death, not much else.
> > > >
> > > > But as a rule of thumb I'd avoid swapping out stuff on
production servers,
> > > > even if it were to SSDs.
> > > > Incidentally the servers I'm talking about here have
their OS and swap on
> > > > Intel DC S3710s (200GB) and the actual storage on plenty of
1.6TB DC
> > > > S3610s.
> > > >
> > > > Relying on the kernel to make swap decisions is likely to
result in much
> > > > reduced performance even with fast SWAP when you're
overcommitting things
> > > > on that scale.
> > > >
> > > >
> > > > But read on.
> > > >
> > > >> Seems that these IMAP processes are long lived processes
(idling most of the time) that don?t need that much of the contents of real
memory available for much of the life of the process. I use a database proxy in
front of MySQL (for my web apps) so that there can be a large number of TCP
connections to the proxy where the frontend requests are queued for execution
using a small number of backend connections.
> > > >>
> > > >> Could Dovecot IMAP be re-written to be more efficient so
it works more like MySQL (or other scalable data servers) that could handle a
million or more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP
sessions aren?t doing much most of the time and shouldn?t really average 2MB of
active data per session that needs to be resident in main memory at all times.
> > > >>
> > > > See IMAP hibernation:
> > > > https://www.mail-archive.com/dovecot at
dovecot.org/msg63429.html <https://www.mail-archive.com/dovecot at
dovecot.org/msg63429.html>
> > > >
> > > > I'm going to deploy/test this in production in about 2
months from now,
> > > > but if you look at the link and the consequent changelog
entries you'll see
> > > > that it has certain shortcomings and bug fixes in pretty
much each release
> > > > after it was introduced.
> > > >
> > > > But this is the correct way to tackle things, not SWAP.
> > > >
> > > > Alas I'm not expecting miracles and if more than 20% of
the IMAP sessions
> > > > here will be hibernated at any given time I'd be
pleasantly surprised.
> > > >
> > > > Because between:
> > > >
> > > > 1. Finding a sensible imap_hibernate_timeout.
> > > >
> > > > 2. Having well behaved clients that keep idling instead of
restarting the
> > > > sequence
(https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/
<https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/>)
> > > >
> > > > 3. Having lots of mobile clients who either get disconnected
(invisible to
> > > > Dovecot) or have aggressive IDLE timers to overcome carrier
NAT timeouts
> > > > (a large mobile carrier here times out idle TCP sessions
after 2 minutes,
> > > > forcing people to use 1 minute IDLE renewals, making 1. up
there a
> > > > nightmare).
> > > >
> > > > 4. Having really broken clients (don't ask, I can't
tell) which open IMAP
> > > > sessions, don't put them into IDLE and thus having them
expire after 30
> > > > minutes.
> > > >
> > > > the pool of eligible IDLE sessions isn't as big as it
could be, in my case
> > > > at least.
> > > >
> > > >> My mail server isn?t that large yet as I haven?t fully
deployed Dovecot outside my own small group yet, but it would be nice if scaling
Dovecot IMAP to millions of users wasn?t limited to 50,000 IMAP sessions on a
server...
> > > >>
> > > >
> > > > Scaling up is nice and desirable from a cost (rack space,
HW) perspective,
> > > > but the scalability of things OTHER than Dovecot as I
pointed out plus
> > > > that little detail of failure domains (do you really want
half of your
> > > > eggs in one basket?) argue for scaling out after a certain
density.
> > > >
> > > > I'm feeling my way there at this time, but expect more
than 100k sessions
> > > > per server to be tricky.
> > > >
> > > > Lastly, when I asked about 500k sessions per server here not
so long ago,
> > > > (
http://www.dovecot.org/list/dovecot/2016-November/106284.html
<http://www.dovecot.org/list/dovecot/2016-November/106284.html> )
> > > > Timo mentioned that he's not aware of anybody doing more
than 50k per
> > > > server, something I got licked already and definitely will
go to 100k
> > > > eventually.
> > > >
> > > > Regards,
> > > >
> > > > Christian
> > > >>> On Feb 10, 2017, at 11:07 AM, Christian Balzer
<chibi at gol.com> wrote:
> > > >>>
> > > >>> On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote:
> > > >>>
> > > >>>>> 1500 IMAP sessions will eat up about 3GB
alone.
> > > >>>>
> > > >>>> Are you saying that Dovecot needs 2MB of
physical memory per IMAP session?
> > > >>>>
> > > >>> That depends on the IMAP session, read the mailbox
size and index size,
> > > >>> etc.
> > > >>> Some are significantly larger:
> > > >>> ---
> > > >>> PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
> > > >>> 1033864 mail 20 0 97600 67m 54m S 0
0.1 0:01.15 imap
> > > >>> ---
> > > >>>
> > > >>> But yes, as somebody who has mailbox servers with
55k+ session the average
> > > >>> is around 1.6MB.
> > > >>>
> > > >>>> If I want to support a max 100,000 IMAP sessions
per server, I should configure the server to have at least 200GBs of SWAP?
> > > >>>>
> > > >>> You will want:
> > > >>
> > > >>> 2. Understanding how to tune Dovecot and more
importantly the overall
> > > >>> system to such a task (see that PID up there?).
> > > >>> 3. Be willing to deal with stuff like top and ps
taking ages to start/run
> > > >>> and others like atop actually killing dovecot
(performance wise, not
> > > >>> literally) when doing their obviously flawed cleanup
on exit. Some things
> > > >>> clearly do NOT scale well.
> > > >>>
> > > >>> My current goal is to have 100k capable servers that
work well, 200k in a
> > > >>> failover scenario, but that won't be particular
enjoyable.
> > > >>>
> > > >>> Christian
> > > >>>
> > > >>>>> On Feb 10, 2017, at 3:58 AM, Christian
Balzer <chibi at gol.com> wrote:
> > > >>>>>
> > > >>>>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M
wrote:
> > > >>>>>
> > > >>>>>> hello
> > > >>>>>>
> > > >>>>>> could somebody with experience let me
know the dovecot config file settings to handle around 1500 simultaneous
connections over pop3 and 1500 connection over imap simultaneously.
> > > >>>>>>
> > > >>>>>
> > > >>>>> Be very precise here, you expect to see 1500
as the result of
> > > >>>>> "doveadm who |grep pop3 |wc -l"?
> > > >>>>>
> > > >>>>> Because that implies an ungodly number of
POP3 connects per second, given
> > > >>>>> the typically short duration of these.
> > > >>>>>
> > > >>>>> 1500 IMAP connections (note that frequently
a client will have more than
> > > >>>>> the INBOX open and thus have more than one
session and thus process on the
> > > >>>>> server) are a much easier proposition,
provided they are of the typical
> > > >>>>> long lasting type.
> > > >>>>>
> > > >>>>> So can you put a number to your expected
logins per second (both protocols)?
> > > >>>>>
> > > >>>>>> my server
> > > >>>>>>
> > > >>>>>> server configuration
> > > >>>>>> hex core processor, 16 gb ram 1 X 600 gb
15 k rpm for main drive and 2 X 2000
> > > >>>>>> gb hdd for data (No raid)
> > > >>>>>>
> > > >>>>> No RAID and no other replication like DRBD?
> > > >>>>> Why would you even bother?
> > > >>>>>
> > > >>>>> How many users/mailboxes in total with what
quota?
> > > >>>>>
> > > >>>>> 1500 IMAP sessions will eat up about 3GB
alone.
> > > >>>>> You will want more memory, simply to keep
all relevant SLAB bits (inodes,
> > > >>>>> dentries) in RAM.
> > > >>>>>
> > > >>>>> If you really have several hundreds
logins/s, you're facing several
> > > >>>>> bottlenecks:
> > > >>>>> 1. Login processes themselves (easily fixed
by high performance mode)
> > > >>>>> 2. Auth processes (that will depend on your
backends, method mostly)
> > > >>>>> 3. Dovecot master process (spawning mail
processes)
> > > >>>>>
> > > >>>>> The later is a single-threaded process, so
it will benefit from a faster
> > > >>>>> CPU core.
> > > >>>>> It can be dramatically improved by enabling
process re-usage, see:
> > > >>>>> http://wiki.dovecot.org/PerformanceTuning
> > > >>>>>
> > > >>>>> However that also means more memory usage.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> Christian
> > > >>>>>
> > > >>>>>>
> > > >>>>>> thanks
> > > >>>>>> rajesh
> > > >>>>>>
> > > >>>>>
> > > >>>>> [snip]
> > > >>>>> --
> > > >>>>> Christian Balzer Network/Systems
Engineer
> > > >>>>> chibi at gol.com Global OnLine
Japan/Rakuten Communications
> > > >>>>> http://www.gol.com/
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Christian Balzer Network/Systems Engineer
> > > >>> chibi at gol.com <mailto:chibi at gol.com>
<mailto:chibi at gol.com <mailto:chibi at gol.com>> Global OnLine
Japan/Rakuten Communications
> > > >>> http://www.gol.com/ <http://www.gol.com/>
<http://www.gol.com/ <http://www.gol.com/>>
> > > >
> > > >
> > > > --
> > > > Christian Balzer Network/Systems Engineer
> > > > chibi at gol.com <mailto:chibi at gol.com> Global
OnLine Japan/Rakuten Communications
> > > > http://www.gol.com/ <http://www.gol.com/>
> > >
> >
> >
>
>
--
Christian Balzer Network/Systems Engineer
chibi at gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/