Hello, I'm migrating from 1.1.16 running in 4 debian lenny servers virtualized with xenserver and 1 core and 5GB of RAM to 2.1.5 running in 4 ubuntu 12.04 servers with 6 cpu cores and 16GB of RAM virtualized with VMWare, but I'm having lots a performance problems. I don't think that virtualization platform could be the problem, because the new servers running in xenserver has the same problems than running in vmware. I have about 70000 user accounts, most of them without real activity (they are students who doesn't read his email or have its account redirected to other provider). I have about 700-1000 concurrent imap connections. I have storage in nfs (nfsv3, the nfs server is a celerra), but indexes are in local filesystems (each server has its own index fs). Mailboxes are in maildir format. Old servers and actual director servers are load balanced with an radware appdirector load balancer (the new backend servers don't need to be balanced because I'm using a director farm) In the old platform I have scenario number 2 described at http://wiki2.dovecot.org/NFS, but in the new ones I have a director proxy directing all connections from each user to the same server (I don't specify any server for the user, director selects it according to the hash algorithm it has). Some doubts I have for the recommended in that url: * mmap_disable: both single and multi server configurations have mmap_disable=yes but in index file section says that you need it if you have your index files stored in nfs. I have it stored locally. Do I need mmap_disable=yes? What it's the best? * dotlock_use_excl: it is set to no in both configurations, but the comment says that it is needed only in nfsv2. Since I have nfs3, I have it set it to yes. * mail_nfs_storage: In single server is set to no, but in multi server it set to yes. Since I have a director in front of my backend server, what is the recommended? With this configuration, when I have a few connections (about 300-400 imap connections) everything is working fine, but when I disconnect the old servers and direct all my users' connections to the new servers I have lot of errors. server loads increments to over 300 points, with a very high io wait. With atop, I could see that of my 6 cores, I have one with almost 100% waiting for i/o and the other with almost 100% idle, but load of the server is very, very high. With the old servers, I have performance problems, access to mail is slow, but it works. But with the new ones it doesn't work at all. Any idea? -- Angel L. Mateo Mart?nez Secci?n de Telem?tica ?rea de Tecnolog?as de la Informaci?n _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
On 20/06/12 11:40, Angel L. Mateo wrote:> Hello, > > I'm migrating from 1.1.16 running in 4 debian lenny servers > virtualized with xenserver and 1 core and 5GB of RAM to 2.1.5 running in > 4 ubuntu 12.04 servers with 6 cpu cores and 16GB of RAM virtualized with > VMWare, but I'm having lots a performance problems. I don't think that > virtualization platform could be the problem, because the new servers > running in xenserver has the same problems than running in vmware. > > I have about 70000 user accounts, most of them without real > activity (they are students who doesn't read his email or have its > account redirected to other provider). I have about 700-1000 concurrent > imap connections. > > I have storage in nfs (nfsv3, the nfs server is a celerra), but > indexes are in local filesystems (each server has its own index fs). > Mailboxes are in maildir format. > > Old servers and actual director servers are load balanced with an > radware appdirector load balancer (the new backend servers don't need to > be balanced because I'm using a director farm) > > In the old platform I have scenario number 2 described at > http://wiki2.dovecot.org/NFS, but in the new ones I have a director > proxy directing all connections from each user to the same server (I > don't specify any server for the user, director selects it according to > the hash algorithm it has). > > Some doubts I have for the recommended in that url: > > * mmap_disable: both single and multi server configurations have > mmap_disable=yes but in index file section says that you need it if you > have your index files stored in nfs. I have it stored locally. Do I need > mmap_disable=yes? What it's the best? > * dotlock_use_excl: it is set to no in both configurations, but the > comment says that it is needed only in nfsv2. Since I have nfs3, I have > it set it to yes. > * mail_nfs_storage: In single server is set to no, but in multi server > it set to yes. Since I have a director in front of my backend server, > what is the recommended? > > With this configuration, when I have a few connections (about > 300-400 imap connections) everything is working fine, but when I > disconnect the old servers and direct all my users' connections to the > new servers I have lot of errors. server loads increments to over 300 > points, with a very high io wait. With atop, I could see that of my 6 > cores, I have one with almost 100% waiting for i/o and the other with > almost 100% idle, but load of the server is very, very high. > > With the old servers, I have performance problems, access to mail > is slow, but it works. But with the new ones it doesn't work at all. > > Any idea? >I forgot attaching my doveconf. -- Angel L. Mateo Mart?nez Secci?n de Telem?tica ?rea de Tecnolog?as de la Informaci?n _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 -------------- next part -------------- # 2.1.5: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-24-generic x86_64 Ubuntu 12.04 LTS auth_cache_size = 20 M auth_cache_ttl = 1 days auth_debug = yes auth_master_user_separator = * auth_verbose = yes default_process_limit = 1000 disable_plaintext_auth = no log_timestamp = %Y-%m-%d %H:%M:%S login_trusted_networks = 155.54.211.176/28 mail_debug = yes mail_fsync = always mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_privileged_group = mail maildir_stat_dirs = yes mdbox_rotate_size = 20 M passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = session=yes dovecot driver = pam } plugin { lazy_expunge = .EXPUNGED/ .DELETED/ .DELETED/.EXPUNGED/ sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +imapflags sieve_max_redirects = 15 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster at um.es service anvil { client_limit = 2003 } service auth { client_limit = 3000 unix_listener auth-userdb { mode = 0666 } } service doveadm { inet_listener { port = 24245 } } service imap { process_limit = 5120 process_min_avail = 6 vsz_limit = 512 M } service lmtp { inet_listener lmtp { port = 24 } process_min_avail = 10 vsz_limit = 512 M } service pop3 { process_min_avail = 6 } ssl = no ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lda { mail_plugins = " sieve" } protocol lmtp { mail_plugins = " sieve" } local 155.54.211.160/27/27 { doveadm_password = ]dWhu5kB }
On Wed, 2012-06-20 at 11:40 +0200, Angel L. Mateo wrote:> * mmap_disable: both single and multi server configurations have > mmap_disable=yes but in index file section says that you need it if you > have your index files stored in nfs. I have it stored locally. Do I need > mmap_disable=yes? What it's the best?mmap_disable is used only for index files, so with local indexes use "no". (If indexes were on NFS, "no" would probably still work but I'm not sure if the performance would be better or worse. Errors would also trigger SIGBUS crashes.)> * dotlock_use_excl: it is set to no in both configurations, but the > comment says that it is needed only in nfsv2. Since I have nfs3, I have > it set it to yes."yes" is ok.> * mail_nfs_storage: In single server is set to no, but in multi server > it set to yes. Since I have a director in front of my backend server, > what is the recommended?With director you can set this to "no".> With this configuration, when I have a few connections (about 300-400 > imap connections) everything is working fine, but when I disconnect the > old servers and direct all my users' connections to the new servers I > have lot of errors.Real errors that show up in Dovecot logs? What kind of errors?> server loads increments to over 300 points, with a > very high io wait. With atop, I could see that of my 6 cores, I have one > with almost 100% waiting for i/o and the other with almost 100% idle, > but load of the server is very, very high.Does the server's disk IO usage actually go a lot higher, or is it simply waiting without doing much of anything? I wonder if this is related to the inotify problems: http://dovecot.org/list/dovecot/2012-June/066474.html Another thought: Since indexes are stored locally, is it possible that the extra load comes simply from building the indexes on the new servers, while they already exist on the old ones?> mail_fsync = alwaysv1.1 did the equivalent of mail_fsync=optimized. You could see if that makes a difference.> maildir_stat_dirs = yesDo you actually need this? It causes unnecessary disk IO and probably not needed in your case.> default_process_limit = 1000Since you haven't enabled high-performance mode for imap-login processes and haven't otherwise changed the service imap-login settings, this means that you can have max. 1000 simultaneous IMAP SSL/TLS connections.