dovecot - Jan 2007 - Performance, dovecot vs tpop3d

Hello, I am new to the list and have spent extensive time researching 
performance suggestions through the list and the 
documentation.  Particularly interesting is the recent performance 
thread related to the upcoming release.

I just upgraded one of our email servers from tpop3d to dovecot 
1.0rc19.  tpop3d has served us well for many years; however, it 
basically has lost all it's support and developers.  My main liking 
of tpop3d was it's speed.  After the upgrade to dovecot, the server 
began to crawl.  This server runs postfix for the delivery agent and 
now dovecot for pop3 and imap.  The box serves roughly 9000 virtual 
maildir mailboxes with roughly 20 pop3 processes active at any given time.

I ran some tests this morning to determine an exact difference 
between running tpop3d and dovecot.
dovecot:  I/O load averages 81%, System load averages 12.00
tpop3d:   I/O load averages 20%, System load averages  0.50

To ensure it wasn't just a fluke of logins, I let each app run for 
several minutes, and did the test multiple times.  The results were 
not any abnormal user usage and definitely related to application performance.

My current settings in dovecot related to performance are:
mmap_diable = yes
mmap_no_write = yes
Modifying these to no seems to decrease performance slightly but only 
a few percent.

I have also played with various other settings, which didn't give me 
any significant performance boost.

I'm eager for the next release which will have the fsync_disable 
option to see if that helps, it certainly looks promising.

One observation where the majority of the load seems to be generated 
from is when a pop3 connection is loading up a very large 
maildir.  The pop3 process seems to remain around using I/O for a 
considerable time, even when the user has no new email.  Perhaps it 
is rebuilding or verifying it's index files?  Here is an example of a 
lsof of the pop3 process with a user who has roughly 17000 
messages.  They check their email every 10 minutes which seems to add 
significant load.  Note the final line, it's reading an individual 
message file, yet the user has no new email since the last login.  If 
i keep lsofing the process, the final file seems to change and keep 
going through many of their mailbox files.  The process takes many 
minutes to complete.  Is it rebuilding the index on every login?

COMMAND   PID    USER   FD   TYPE    DEVICE    SIZE      NODE NAME
pop3    15148 postfix  cwd    DIR       9,2     152    850160 
/home/domain.net/brock
pop3    15148 postfix  rtd    DIR       9,2     816         2 /
pop3    15148 postfix  txt    REG       9,2 1888690   7096106 
/usr/local/libexec/dovecot/pop3
pop3    15148 postfix  mem    REG       9,2  107724    182934 /lib/ld-2.3.2.so
pop3    15148 postfix  mem    REG       9,2 1578228    182936 
/lib/tls/libc-2.3.2.so
pop3    15148 postfix  mem    REG       9,2   16312     88977 
/lib/libdl-2.3.2.so
pop3    15148 postfix    0u  IPv4 873280915               TCP 
domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT)
pop3    15148 postfix    1u  IPv4 873280915               TCP 
domain.net:pop3->dsl.dyn.06-99.uslogin.net:40950 (CLOSE_WAIT)
pop3    15148 postfix    2w  FIFO       0,5         873280916 pipe
pop3    15148 postfix    3r   CHR       1,9             27963 /dev/urandom
pop3    15148 postfix    4r  FIFO       0,5         873281456 pipe
pop3    15148 postfix    5w  FIFO       0,5         873281456 pipe
pop3    15148 postfix    6r  FIFO       0,5         873281457 pipe
pop3    15148 postfix    7w  FIFO       0,5         873281457 pipe
pop3    15148 postfix    8u   REG       9,2  205740   7272682 
/var/spool/mail/domain.net/brock/dovecot.index
pop3    15148 postfix    9u   REG       9,2   15020   7225137 
/var/spool/mail/domain.net/brock/dovecot.index.log
pop3    15148 postfix   10u   REG       9,2 1585152   7290906 
/var/spool/mail/domain.net/brock/dovecot.index.cache
pop3    15148 postfix   11r   REG       9,2    1992   5040734 
/var/spool/mail/domain.net/brock/cur/1165164651.V902I4cea5e.s3.visp.net:2,S

I'm eager to solve this new I/O issue which has cropped up after 
upgrading to dovecot.  Despite the severe performance degradation, I 
feel dovecot has awesome potential and is feature rich, and most 
importantly it's being maintained, so I want to stick with it!

Thank you for any help offered!  I'm hopeful for any suggestions to 
solve this performance issue or diagnose it further to determine the cause.

- Nate

Hi..

wow.. this is the first time I can recall someone complaining Dovecot is so much
SLOWER than their old install.  :P

Nate wrote:
> files.  The process takes many minutes to complete.  Is it rebuilding 
> the index on every login?
The only case I can see where Dovecot would rebuild indexes on _every_ connect 
would be if you had INDEX=MEMORY in your mail_location.

--
Curtis Maloney
cmaloney at cardgate.net

>> 2.  logout, log back in immediatly, there is no re-indexing and the
>> list is available in < 1 second.
>> 3.  logout, log back in 12 minutes later.  The list is returned in
>> apx 15 seconds and again in a lsof I see pop3 accessing files inside
>> the cur/ directory.
...
>
> Of added note, when running a stat on the index files (dovecot.index,
> dovecot.index.cache, dovecot.index.log), the Modify and Change
> parameters remain static (no changes).  So if it's rebuilding
> anything it's doing it in memory and not modifying the index files.
>
> If I deliver a new message to the account, it does rebuild the files
> for sure and increments the Modify and Change params.
Perhaps what you might be seeing is file system caching.  I notice this
when performing NFS I/O tests: the first re-read is fast but decays
over time as data is rolled out of the cache.

Joseph Tam <tam at math.ubc.ca>

At 09:01 PM 1/30/2007, Curtis Maloney wrote:
>Nate wrote:
>>Perhaps another test i'll run, just to compare apples to apples 
>>tomorrow would be to use a memory index with dovecot as well, and 
>>see if that drops the I/O load to the same as when tpop3d is running.
>
>Speaking of "Apples to Apples", you might want to try with 
>INDEX=MEMORY on Dovecot, see how that fares.
>
>If your users are only POP3 and download all messages at once, not 
>keeping mail on the server, you can (iirc) disable indexing 
>altogether, since it's a wasted effort.
That test was on my list to try out today.  I ran the test, and 
performance was severely degraded.  Upon every new login, the entire 
index was rebuilt.  So with any login it takes the full ~30 seconds 
to rebuild the index on a 17000 email user.

I've confirmed also that tpop3d is doing no indexing whatsoever on 
disk.  It isn't even compiled with that option.  So it seems it is 
probably disk caching which is assisting tpop3d with returning a 2nd 
login (within a few minutes) so quickly.

I wish it were the case that all users downloaded messages, but it 
seems all too often users click the 'leave messages on server', and 
never expire them.  We have quite a few users (thousands probably) in 
that condition.  We occasionally delete old stuff.  Did I hear about 
an expire plugin upcoming?  That would be useful.

- Nate

<snip>

What version of dovecot?

Can you post your entire dovecot.conf?