thr3ads.net - dovecot - [Dovecot] POP3 Performance [Mar 2012]

If this information is useful, please help other people find it:
Share via:

Mauricio López Riffo

2012-Mar-16 11:07 UTC

[Dovecot] POP3 Performance

Hi,

     We actually have a mail hosting solutions with aprox. 100 thousand 
of email account, where about 90% of a customers use POP3 like email 
configuration.  About a few mounths (we perfomed a lot of migration 
throught mbox email software to Maildir with dovecot) but i can see that 
the performance is very poor and receive complaint about delays of 
autentications of accounts.

The solution lives in Metrocluster Netapp storage, filesystem NFS, 
VMware as a virtualization (the mtas are a virtual machines lives in 
netapp too) about 4T of data mails and a 10G network connection (betwen 
mtas and nfs storage)  All account information work in LDAP plataform 
(two servers in replicated mode, no high average or delays detected in 
this servers)

When the traffic have a peak of 1800 concurrent connections POP3, all of 
service suffer a high load average (about 8 - 20 load average in each 
dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, 
autenticacion takes about 60 miliseconds)

Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 
6G RAM (virtual machine) and share's hardware with a exim instance, like 
a MTA relay system (autenticated relay)

Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of 
bandwith)

Attach of dovecot -n output:

# 2.0.18: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final)
auth_debug_passwords = yes
auth_default_realm = portalplata.cl
auth_realms = portalplata.cl
auth_verbose = yes
auth_verbose_passwords = plain
auth_worker_max_count = 100
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot.log
default_process_limit = 200
default_vsz_limit = 512 M
disable_plaintext_auth = no
first_valid_gid = 12
first_valid_uid = 8
lock_method = dotlock
login_greeting = Dovecot mta10
mail_cache_min_mail_count = 5
mail_debug = yes
mail_fsync = always
mail_full_filesystem_access = yes
mail_gid = 12
mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u
mail_nfs_storage = yes
mail_plugins = " quota"
mail_uid = 8
maildir_copy_with_hardlinks = no
passdb {
   args = /etc/dovecot/dovecot-ldap.conf.ext
   driver = ldap
}
plugin {
   mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename
   mail_log_fields = box from subject
   quota = maildir
}
postmaster_address = mail at mail.com
protocols = imap pop3
sendmail_path = /usr/lib/sendmail
service auth {
   unix_listener auth-userdb {
     mode = 0600
     user = exim
   }
}
service imap-login {
   service_count = 0
}
service imap-postlogin {
   executable = script-login /usr/local/bin/postlogin.sh
   user = root
}
service imap {
   executable = imap imap-postlogin
}
service pop3-login {
   inet_listener pop3s {
     port = 995
     ssl = yes
   }
   service_count = 0
}
service pop3 {
   process_limit = 1024
}
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_key = </etc/pki/dovecot/private/dovecot.pem
userdb {
   args = /etc/dovecot/dovecot-ldap.conf.ext
   driver = ldap
}
verbose_proctitle = yes
protocol lmtp {
   mail_plugins = " quota"
}
protocol lda {
   mail_plugins = " quota"
}
protocol imap {
   imap_capability    mail_max_userip_connections = 10
   mail_plugins = " quota autocreate notify quota imap_quota mail_log"
   plugin {
     autocreate = Sent Items
     autocreate2 = Spam
     autocreate3 = Drafts
     autocreate4 = Trash
     autosubscribe = Sent Items
     autosubscribe2 = Spam
     autosubscribe3 = Drafts
     autosubscribe4 = Trash
   }
}
protocol pop3 {
   mail_plugins = " quota quota notify mail_log"
   pop3_reuse_xuidl = yes
   pop3_save_uidl = yes
   pop3_uidl_format = %u
}


Output of account information in a LDAP:

# nettester, email.net, MAIL, USERS, cl
dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl
dc: email.net
mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir
uid: nettester
cn: nettester at email.net
sn: nettester at email.net
gidNumber: 12
homeDirectory: /export/mdir/3/12/nettester at email.net
mail: nettester at email.net
uidNumber: 8
objectClass: mailUser
objectClass: posixAccount
objectClass: mailSetting
loginShell: /bin/false
description: enable
service: pop3
service: imap
service2: webmail
mailRate: 200
mailQuota: 1024M
deliveryMode: none
mailReplyText: .

Any suggestions? All ideas will be have a good receptions ;)


Pd: Sorry my english

-- 
Mauricio L?pez Riffo
Red Hat Certified Engineer
804006455319519
Administrador de Servicios Internet
?rea Ingenier?a
Gtd Internet S.A.
http://www.grupogtd.com/
Moneda 920, Oficina 602 - Fono : +562 4139742

Robert Schetterer

2012-Mar-16 12:10 UTC

head link

[Dovecot] POP3 Performance

Am 16.03.2012 12:07, schrieb Mauricio L?pez Riffo:> Hi,
> 
>     We actually have a mail hosting solutions with aprox. 100 thousand
> of email account, where about 90% of a customers use POP3 like email
> configuration.  About a few mounths (we perfomed a lot of migration
> throught mbox email software to Maildir with dovecot) but i can see that
> the performance is very poor and receive complaint about delays of
> autentications of accounts.
> 
> The solution lives in Metrocluster Netapp storage, filesystem NFS,
> VMware as a virtualization (the mtas are a virtual machines lives in
> netapp too) about 4T of data mails and a 10G network connection (betwen
> mtas and nfs storage)  All account information work in LDAP plataform
> (two servers in replicated mode, no high average or delays detected in
> this servers)
> 
> When the traffic have a peak of 1800 concurrent connections POP3, all of
> service suffer a high load average (about 8 - 20 load average in each
> dovecot) and authenticacion takes about 2 -10 seconds (in low traffic,
> autenticacion takes about 60 miliseconds)
> 
> Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with
> 6G RAM (virtual machine) and share's hardware with a exim instance,
like
> a MTA relay system (autenticated relay)
> 
> Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of
> bandwith)
> 
> Attach of dovecot -n output:
> 
> # 2.0.18: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final)
> auth_debug_passwords = yes
> auth_default_realm = portalplata.cl
> auth_realms = portalplata.cl
> auth_verbose = yes
> auth_verbose_passwords = plain
> auth_worker_max_count = 100
> base_dir = /var/run/dovecot/
> debug_log_path = /var/log/dovecot.log
> default_process_limit = 200
> default_vsz_limit = 512 M
> disable_plaintext_auth = no
> first_valid_gid = 12
> first_valid_uid = 8
> lock_method = dotlock
> login_greeting = Dovecot mta10
> mail_cache_min_mail_count = 5
> mail_debug = yes
> mail_fsync = always
> mail_full_filesystem_access = yes
> mail_gid = 12
> mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u
> mail_nfs_storage = yes
> mail_plugins = " quota"
> mail_uid = 8
> maildir_copy_with_hardlinks = no
> passdb {
>   args = /etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> plugin {
>   mail_log_events = delete undelete expunge copy mailbox_delete
> mailbox_rename
>   mail_log_fields = box from subject
>   quota = maildir
> }
> postmaster_address = mail at mail.com
> protocols = imap pop3
> sendmail_path = /usr/lib/sendmail
> service auth {
>   unix_listener auth-userdb {
>     mode = 0600
>     user = exim
>   }
> }
> service imap-login {
>   service_count = 0
> }
> service imap-postlogin {
>   executable = script-login /usr/local/bin/postlogin.sh
>   user = root
> }
> service imap {
>   executable = imap imap-postlogin
> }
> service pop3-login {
>   inet_listener pop3s {
>     port = 995
>     ssl = yes
>   }
>   service_count = 0
> }
> service pop3 {
>   process_limit = 1024
> }
> ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
> ssl_key = </etc/pki/dovecot/private/dovecot.pem
> userdb {
>   args = /etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> verbose_proctitle = yes
> protocol lmtp {
>   mail_plugins = " quota"
> }
> protocol lda {
>   mail_plugins = " quota"
> }
> protocol imap {
>   imap_capability >   mail_max_userip_connections = 10
>   mail_plugins = " quota autocreate notify quota imap_quota
mail_log"
>   plugin {
>     autocreate = Sent Items
>     autocreate2 = Spam
>     autocreate3 = Drafts
>     autocreate4 = Trash
>     autosubscribe = Sent Items
>     autosubscribe2 = Spam
>     autosubscribe3 = Drafts
>     autosubscribe4 = Trash
>   }
> }
> protocol pop3 {
>   mail_plugins = " quota quota notify mail_log"
>   pop3_reuse_xuidl = yes
>   pop3_save_uidl = yes
>   pop3_uidl_format = %u
> }
> 
> 
> Output of account information in a LDAP:
> 
> # nettester, email.net, MAIL, USERS, cl
> dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl
> dc: email.net
> mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir
> uid: nettester
> cn: nettester at email.net
> sn: nettester at email.net
> gidNumber: 12
> homeDirectory: /export/mdir/3/12/nettester at email.net
> mail: nettester at email.net
> uidNumber: 8
> objectClass: mailUser
> objectClass: posixAccount
> objectClass: mailSetting
> loginShell: /bin/false
> description: enable
> service: pop3
> service: imap
> service2: webmail
> mailRate: 200
> mailQuota: 1024M
> deliveryMode: none
> mailReplyText: .
> 
> Any suggestions? All ideas will be have a good receptions ;)
> 
> 
> Pd: Sorry my english
> 
looks like you need to play with some config stuff and do more debug on
your possible bottlenecks, what did you allready played with dovecot to
high performance

guess Timo will help about config settings

after all for short
to read

http://wiki.dovecot.org/Authentication/Caching
http://wiki.dovecot.org/LoginProcess
http://wiki2.dovecot.org/Services

any reasons for that

mail_full_filesystem_access = yes ?

general

nfs may not optimal, but that a long story
also virtual machines have their pros and contras
maildir is not so good in performance
-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

Jan-Frode Myklebust

2012-Mar-16 12:11 UTC

head link

[Dovecot] POP3 Performance

One quick fix to try, if it's the login-time that's killing you, is to
enable
auth caching:

	http://wiki2.dovecot.org/Authentication/Caching

that should offload your backend LDAP-servers from doing bind() on ever
login, had a huge login performance impact for us. We use
"auth_cache_size = 100 M", which gives us 99% cache hits:

	dovecot: auth: Authentication cache hits 3654591/3669119 (99%)
	dovecot: auth: Authentication cache inserts: positive: 588030 80931909B,
negative: 912 49888B


   -jf

Steve Campbell

2012-Mar-16 12:54 UTC

head link

[Dovecot] POP3 Performance

On 3/16/2012 7:07 AM, Mauricio L?pez Riffo wrote:> Hi,
>
>     We actually have a mail hosting solutions with aprox. 100 thousand 
> of email account, where about 90% of a customers use POP3 like email 
> configuration.  About a few mounths (we perfomed a lot of migration 
> throught mbox email software to Maildir with dovecot) but i can see 
> that the performance is very poor and receive complaint about delays 
> of autentications of accounts.
>
> The solution lives in Metrocluster Netapp storage, filesystem NFS, 
> VMware as a virtualization (the mtas are a virtual machines lives in 
> netapp too) about 4T of data mails and a 10G network connection 
> (betwen mtas and nfs storage)  All account information work in LDAP 
> plataform (two servers in replicated mode, no high average or delays 
> detected in this servers)
>
> When the traffic have a peak of 1800 concurrent connections POP3, all 
> of service suffer a high load average (about 8 - 20 load average in 
> each dovecot) and authenticacion takes about 2 -10 seconds (in low 
> traffic, autenticacion takes about 60 miliseconds)
>
> Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 
> 6G RAM (virtual machine) and share's hardware with a exim instance, 
> like a MTA relay system (autenticated relay)
>
> Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of 
> bandwith)
>
> Attach of dovecot -n output:
>
> # 2.0.18: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final)
> auth_debug_passwords = yes
> auth_default_realm = portalplata.cl
> auth_realms = portalplata.cl
> auth_verbose = yes
> auth_verbose_passwords = plain
> auth_worker_max_count = 100
> base_dir = /var/run/dovecot/
> debug_log_path = /var/log/dovecot.log
> default_process_limit = 200
> default_vsz_limit = 512 M
> disable_plaintext_auth = no
> first_valid_gid = 12
> first_valid_uid = 8
> lock_method = dotlock
> login_greeting = Dovecot mta10
> mail_cache_min_mail_count = 5
> mail_debug = yes
> mail_fsync = always
> mail_full_filesystem_access = yes
> mail_gid = 12
> mail_location = 
> maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u
> mail_nfs_storage = yes
> mail_plugins = " quota"
> mail_uid = 8
> maildir_copy_with_hardlinks = no
> passdb {
>   args = /etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> plugin {
>   mail_log_events = delete undelete expunge copy mailbox_delete 
> mailbox_rename
>   mail_log_fields = box from subject
>   quota = maildir
> }
> postmaster_address = mail at mail.com
> protocols = imap pop3
> sendmail_path = /usr/lib/sendmail
> service auth {
>   unix_listener auth-userdb {
>     mode = 0600
>     user = exim
>   }
> }
> service imap-login {
>   service_count = 0
> }
> service imap-postlogin {
>   executable = script-login /usr/local/bin/postlogin.sh
>   user = root
> }
> service imap {
>   executable = imap imap-postlogin
> }
> service pop3-login {
>   inet_listener pop3s {
>     port = 995
>     ssl = yes
>   }
>   service_count = 0
> }
> service pop3 {
>   process_limit = 1024
> }
> ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
> ssl_key = </etc/pki/dovecot/private/dovecot.pem
> userdb {
>   args = /etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> verbose_proctitle = yes
> protocol lmtp {
>   mail_plugins = " quota"
> }
> protocol lda {
>   mail_plugins = " quota"
> }
> protocol imap {
>   imap_capability >   mail_max_userip_connections = 10
>   mail_plugins = " quota autocreate notify quota imap_quota
mail_log"
>   plugin {
>     autocreate = Sent Items
>     autocreate2 = Spam
>     autocreate3 = Drafts
>     autocreate4 = Trash
>     autosubscribe = Sent Items
>     autosubscribe2 = Spam
>     autosubscribe3 = Drafts
>     autosubscribe4 = Trash
>   }
> }
> protocol pop3 {
>   mail_plugins = " quota quota notify mail_log"
>   pop3_reuse_xuidl = yes
>   pop3_save_uidl = yes
>   pop3_uidl_format = %u
> }
>
>
> Output of account information in a LDAP:
>
> # nettester, email.net, MAIL, USERS, cl
> dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl
> dc: email.net
> mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir
> uid: nettester
> cn: nettester at email.net
> sn: nettester at email.net
> gidNumber: 12
> homeDirectory: /export/mdir/3/12/nettester at email.net
> mail: nettester at email.net
> uidNumber: 8
> objectClass: mailUser
> objectClass: posixAccount
> objectClass: mailSetting
> loginShell: /bin/false
> description: enable
> service: pop3
> service: imap
> service2: webmail
> mailRate: 200
> mailQuota: 1024M
> deliveryMode: none
> mailReplyText: .
>
> Any suggestions? All ideas will be have a good receptions ;)
>
>
> Pd: Sorry my english
It doesn't seem to matter what type of hardware you might have, NFS can 
cause real bottlenecks, even to the point that your machine may report 
disk errors.

Unfortunately, it's an evil necessity in some shops, but any way to 
eliminate NFS when large throughput is occurring will definitely help. 
Make sure you're running the latest version of NFS on all machines since 
V3 and V4 don't always like each other.

I don't have a solution for it's replacement other than expensive 
hardware solutions.

steve

Timo Sirainen

2012-Mar-16 13:07 UTC

head link

[Dovecot] POP3 Performance

On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote:
>    We actually have a mail hosting solutions with aprox. 100 thousand of
email account, where about 90% of a customers use POP3 like email configuration.
About a few mounths (we perfomed a lot of migration throught mbox email software
to Maildir with dovecot) but i can see that the performance is very poor and
receive complaint about delays of autentications of accounts.
> 
> The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware
as a virtualization (the mtas are a virtual machines lives in netapp too) about
4T of data mails and a 10G network connection (betwen mtas and nfs storage)  All
account information work in LDAP plataform (two servers in replicated mode, no
high average or delays detected in this servers)
Maildir isn't very good for POP3, especially if the POP3 clients delete the
mails after download. With Dovecot you could look into switching to multi-dbox
format, which would have much better performance.
> When the traffic have a peak of 1800 concurrent connections POP3, all of
service suffer a high load average (about 8 - 20 load average in each dovecot)
and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion
takes about 60 miliseconds)
What does the CPU usage and NFS IOPS usage look like during those times? Meaning
is the problem related to disk usage or something else?

Note that for POP3 connections you don't get the "OK Logged in"
reply until all of the message sizes have been read into memory.
> Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G
RAM (virtual machine) and share's hardware with a exim instance, like a MTA
relay system (autenticated relay)
Are you randomly redirecting users to different Dovecot servers? Dovecot
director would work better: http://wiki2.dovecot.org/Director
> auth_worker_max_count = 100
Auth workers are irrelevant with LDAP.
> lock_method = dotlock
fcntl would be faster, if your NFS setup can handle it.
> mail_cache_min_mail_count = 5
I'm not really sure if it's a good idea to ever set this anything else
than 0. Of course if you have detected that this actually decreases disk IO
I'd be interested to see numbers.
> mail_full_filesystem_access = yes
If your users are sharing the same UID, this means all the users can access each
others' mails now! Even if they have different UIDs this is unlikely to be
helpful.
> mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u
> mail_nfs_storage = yes
Is /data also on NFS? Or does each server have its own local indexes?
> maildir_copy_with_hardlinks = no
This makes IMAP COPY command slower. Is there a reason why you've disabled
it?
> service imap-postlogin {
>  executable = script-login /usr/local/bin/postlogin.sh
>  user = root
> }
post-login script makes logins slower. What do you do in it? Why only for IMAP,
not POP3?
>  plugin {
>    autocreate = Sent Items
>    autocreate2 = Spam
>    autocreate3 = Drafts
>    autocreate4 = Trash
>    autosubscribe = Sent Items
>    autosubscribe2 = Spam
>    autosubscribe3 = Drafts
>    autosubscribe4 = Trash
>  }
These also slow logins down a little bit. v2.1 fixes that.
> protocol pop3 {
>  mail_plugins = " quota quota notify mail_log"
>  pop3_reuse_xuidl = yes
>  pop3_save_uidl = yes
>  pop3_uidl_format = %u
> }
pop3_uidl_format=%u is a little bad, since it doesn't include %v. And you
can improve pop3 performance with:

pop3_no_flag_changes=yes

And if the maildir filenames don't contain S=1234 sizes, this also makes a
huge difference:

pop3_fast_size_lookups=yes

Mark Alan

2012-Mar-17 14:14 UTC

head link

[Dovecot] POP3 Performance

On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen <tss at iki.fi>
wrote:> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote:
> pop3_no_flag_changes=yes
Is it the same as pop3_no_flag_updates=yes ?


M.

Timo Sirainen

2012-Mar-17 14:40 UTC

head link

[Dovecot] POP3 Performance

On 17.3.2012, at 16.14, Mark Alan wrote:
> On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen <tss at iki.fi>
wrote:
>> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote:
> 
>> pop3_no_flag_changes=yes
> 
> Is it the same as pop3_no_flag_updates=yes ?
Yeah. I wrote it from my memory.

Possibly Parallel Threads

Search for more apparently analagous threads

dovecot - Mar 2012 - POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

[Dovecot] POP3 Performance

Possibly Parallel Threads