dovecot - May 2009 - Degeneration of CPU Performance

Hello everybody!

    We have a interesting issue about dovecot behavior here. First, the 
scenario:
    We have 2 server running with the same load, one with our old pop3 
solution (out of date) and other with Dovecot. We realized that dovecot 
are comsuming more CPU, and this consumption is growing day by day. When 
we starts dovecot, it runs between 40%-45% of CPU consumption and our 
old solution runs on 30-35%. This is quite acceptable, so no problem 
here. The problem is one day after it jumps to 45%-55% of cpu 
comsunption while the old pop3 solution runs on the same CPU consumption 
of one day before (30%-35). I attached a graph with this information.

In this graph the green line is dovecot CPU consumption and the blue 
line is the old solution.

In this graph we restart dovecot at monday morning. This restart is 
noticed as a big fall of CPU consumption(green line) in the graph. After 
that, it stay between 40%-45%, one day after it runs on 45%-55% and next 
day, it reaches 50%-60%! You can notice that the blue line (old pop3 
solution) runs with the same behavior, so we have the same load. We have 
other monitoring informations that confirm it.

I looked for some bugs about this, but I didn't find anything. The 
question is:
Is there any dovecot problem or wrong configuration that could cause 
this degeneration of CPU performance? Any sugestion about what can be 
the cause of this?

I attached dovecot -n output. Two important information: we store the 
mailboxes in NFS and we are only using pop3 protocol.


Thanks in advance!

-- 
Thiago Monaco Papageorgiou <thiago.monaco at corp.terra.com.br>
________________________
Terra Networks Brasil S/A 
Tel: +55 (51) 3284-4274

-------------- next part --------------
A non-text attachment was scrubbed...
Name: grapher.jpeg
Type: image/jpeg
Size: 42306 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090507/72517023/attachment-0002.jpeg>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dovecot_n
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090507/72517023/attachment-0002.pl>

Thiago Monaco Papageorgiou wrote:
>    We have 2 server running with the same load, one with our old pop3 
> solution (out of date) and other with Dovecot. We realized that dovecot 
> are comsuming more CPU, and this consumption is growing day by day. When 
> we starts dovecot, it runs between 40%-45% of CPU consumption and our 
> old solution runs on 30-35%. This is quite acceptable, so no problem 
> here. The problem is one day after it jumps to 45%-55% of cpu 
> comsunption while the old pop3 solution runs on the same CPU consumption 
> of one day before (30%-35). I attached a graph with this information.
Are your using leaving mail on the server?  If not, you may find it 
advantageous to disable the indexing, since it's of no real use for
"drive
by collect" mail.
> In this graph the green line is dovecot CPU consumption and the blue 
> line is the old solution.
> 
> In this graph we restart dovecot at monday morning. This restart is 
> noticed as a big fall of CPU consumption(green line) in the graph. After 
> that, it stay between 40%-45%, one day after it runs on 45%-55% and next 
> day, it reaches 50%-60%! You can notice that the blue line (old pop3 
> solution) runs with the same behavior, so we have the same load. We have 
> other monitoring informations that confirm it.
This, to me, is consistent with Dovecot spending a lot of time indexing the 
overnight deliveries as everyone logs on in the morning.

Are you using dovecot deliver as your LDA?
> Is there any dovecot problem or wrong configuration that could cause 
> this degeneration of CPU performance? Any sugestion about what can be 
> the cause of this?
It could be that Dovecot is doing work you don't need it to.  By which I 
mean building indices.
> I attached dovecot -n output. Two important information: we store the 
> mailboxes in NFS and we are only using pop3 protocol.
As I said above, if your users are using this service ONLY to collect mail, 
not to store it, then the indexes Dovecot tries to maintain are a waste of 
effort.

You can read more about POP3 configuration in the wiki at: 
http://wiki.dovecot.org/POP3Server

The MailLocation page also has some notes about index file placement, including:

"""
If you really want to, you can also disable the index files completely by 
appending :INDEX=MEMORY.
"""

-- 
Curtis Maloney
cmaloney at cardgate.net

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 7 May 2009, Thiago Monaco Papageorgiou wrote:
> I looked for some bugs about this, but I didn't find anything. The
question
v1.1.6 is pretty old, though.
> I attached dovecot -n output. Two important information: we store the 
> mailboxes in NFS and we are only using pop3 protocol.
What's hashdir? Maybe this is a performance killer?

Did you checked things like memory consumption and if there are processes 
which do consume lots of CPU, e.g. via "top" or similiar command.
Maybe
your server spends the time on the network?

Bye,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBSgPba3WSIuGy1ktrAQLNvAgAxV3Kb5j1SzihB318ECzvLEN1gazzptD8
E0ie735sCL2B4xpD/Iz1MlJbBbvDBoGvYt67Bn/DKpZ0lWF73frw3D3ITvvssGSW
J6cSfc6jDbaejwDPrUAByJyZkHyBpSQSBFdyvBWBfwHPeQn4YUFcyHLQW7O+9BDg
Ld9oh/2ALmhwAC3Rt7CAgLcS0XNyKolJY+kpUgoW/V8BKXVley6ud4nqlAVuFGsL
LSdKNof2MNEU1zWaD16GenXx6RiDLvbdXvZmZbROYRCXolj+gICwORY8s/o8GYUC
LvXXPwKCnMYBW6ZD4dvHE8K9vwL/mw8qMZpdRtMbbMGhA5qz9TugGw==ga6+
-----END PGP SIGNATURE-----

On Thu, 2009-05-07 at 15:40 -0300, Thiago Monaco Papageorgiou
wrote:
> Hello everybody!
> 
>     We have a interesting issue about dovecot behavior here. First, the 
> scenario:
>     We have 2 server running with the same load, one with our old pop3 
> solution (out of date) and other with Dovecot. We realized that dovecot 
> are comsuming more CPU, and this consumption is growing day by day. When 
> we starts dovecot, it runs between 40%-45% of CPU consumption and our 
> old solution runs on 30-35%. This is quite acceptable, so no problem 
> here. The problem is one day after it jumps to 45%-55% of cpu 
> comsunption while the old pop3 solution runs on the same CPU consumption 
> of one day before (30%-35). I attached a graph with this information.
I see you're using NFS and Linux. We've seen something similar.

Try to find out where this CPU time is being spent - in the kernel, or
in userland. 'top' will tell you, just start it, and look at the second
or 3rd line where it says 'CPU'. 'us' is user time, 'sy'
is system time,
and 'si' is 'system interrupt'. The latter two are time spent in
the
kernel.

If all CPU is used by 'us' then it's really dovecot that is eating
cycles. If it is 'sy' or 'si' it's the kernel.

In that case, you might want to upgrade. Upgrade to 2.6.27.10 at least,
2.7.27.latest preferably - many NFS bugs have been fixed there. I have
no idea if there is a drop-in CentOS kernel >= 2.6.27.10 - you might
have to compile your own kernel.

Mike.