Hi
We've been using Dovecot as our primary IMAP and POP3 server for around
70,000 users for a couple of months now. On the whole, we've been very
pleased with the performance of Dovecot - and are extremely grateful to the
developers! However, we have had a few issues, some of which we have yet to
resolve.
We chose Dovecot because it was the only IMAP/POP3 server we could find that
supported direct mysql authentication and both mboxes and maildirs. With
exim as our main SMTP server, we were able to convert the inboxes and imap
folders for all 70,000 users from mboxes to maildirs without any downtime
whatsoever!
For those who are interested, we have a cluster of 5 web/imap servers
(running our own webmail software) and a further 5 pop servers all placed
behind 2 Zeus load balancers and running RHEL 4 (actually CentOS 4.1).
Storage is provided by an EMC Clariion device, which is mounted via NFS onto
all servers in the cluster over gigabit ethernet.
1. index files
The main problem has always been the index files becoming corrupted. This
seems to have improved with the Alpha 4 release, but still happens for
several users each day. Perhaps it would be possible for Dovecot to just
delete index files and re-create them when they are corrupted, rather than
just erroring? Also, how do you turn off index files altogether? Even when
mail_never_cache_fields is set for all available fields, they still get
created.
Another issue relating to the index files is that, if a user is deleted and
then another user created with the same name, dovecot doesn't have
permissions to re-write the index files, because it has a different uid. So,
whenever a user is deleted, we have to delete the index files from every
machine in the cluster. We could store index files on the NFS device, but
this would affect performance (and might cause locking problems). A better
solution would be to make the uid one of the variables available in
default_mail_env. By naming index files by uid rather than username, this
wouldn't be an issue.
2. initgroups()
We use an nss-mysql to store all non-administrative system users in a mysql
database. We often encounter problems with applications that use the
initgroups() function, since this returns all users and groups - which in
our case returns masses of data from mysql. When using mysql (or ldap etc)
for authentication, it would be useful if there were an option to prevent
additional system lookups. At present, we have to comment out the following
in /src/lib/restrict-access.c, or the server load goes through the roof:
if (initgroups(env, gid) != 0) {
i_fatal("initgroups(%s, %s) failed: %m", env, dec2str(gid));
}
3. base_dir permissions bug
Since the alpha 4 release, it seems that the permissions dovecot
automatically sets for the base_dir are not sufficient to allow the
authentication user to access sockets, unless this user belongs to the same
group as the login user - which is contrary to the instructions in the
documentation. I'm pretty sure this is a bug, but perhaps someone could
confirm.
4. authentication caching
Also since the alpha 4 release, we have found that, once the authentication
cache is full, all subsequent login attempts for users that haven't been
cached return "password mismatch". I though this might be a conflict
with
nscd, but it happens whether nscd is running or not. So, for the time being,
we have had to disable the authentication cache.
I'd be very keen to hear from anyone who has any feedback on any of the
points above.
Thanks
Marcus
--
Marcus Don
Applications Development Manager
Namesco Limited
Main Line: +44 (0)870 120 8888
Main Fax: +44 (0)870 120 8008
Tech Support: +44 (0)870 162 4950
Email: mdon@names.co.uk
Website: http://www.names.co.uk
Address: Acton House, Perdiswell Park, Worcester, WR3 7GD
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender
immediately. If you are not the intended recipient you are notified that
disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. Please note that any
views or opinions presented in this email are solely those of the author and
do not necessarily represent those of the company. Finally, the recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.
------------------------------------------------------------------------
---------------------------------------
| Free Dial-up - no need to register! |
---------------------------------------
Namesco Limited provides Free dialup access to anyone that wishes to use it.
You do not even have to be a customer, there is no need to register and is
especially handy if you are unable to connect with your current provider.
Simply use the following details:
Dialup Number: 0844 535 2100*
Username: namesco
Password: easydialup
* Local call rate charges apply, 33% Cheaper than BT's 0845 local call rate.
If you need any help in configuring the Free Dial-up service on your
computer, please do not hesitate to call us on 0870 162 4950.
Marcus Don wrote: > Also, how do you turn off index files altogether? Put "INDEX=MEMORY" on your default_mail_env, by this you make Dovecot to use non-persistent memory indexes, for example: default_mail_env = maildir:/nfs/%d/%u/Maildir:INDEX=MEMORY> So, whenever a user is deleted, we have to delete the index files from every > machine in the cluster. A better solution would be to make the uid one of > the variables available in default_mail_env. By naming index files by uid > rather than username, this wouldn't be an issue.If you move to memory indexes you have no use for this but since you use SQL authentication backend you could have very flexible layout for your mail spool and indexes. Just return properly formatted "mail" field on user_query, this will overwrite your default_mail_env you have in dovecot.conf. For example user_query = SELECT uid, .., concat('maildir:/nfs/', domain, '/', uid, '/Maildir:INDEX=/var/index/', domain, '/' uid) AS mail, .. would read mail from /nfs/%domain/%uid/Maildir and store indexes to /var/index/%domain/%uid Cool but not very neat. More clean way to do this would be to have templates for environment variables like home, mail and index and use field names from SQL as fill in variables, like: userdb sql { set home = /nfs/%%spool%%/%d/%%uid%% set mail = maildir:/nfs/%%spool%%/%d/%%uid%%/mail set index = /var/%3RHu/%d/%%uid%% args = /opt/dovecot/etc/sql.conf } and then get fields like "spool" and "uid" from SQL and let the Dovecot to fill them into template environments you defined. I know that Timo has more than enough work at the moment so if anyone on the list have some spare time please feel free to take a look on this :) > 4. authentication caching> > Also since the alpha 4 release, we have found that, once the authentication > cache is full, all subsequent login attempts for users that haven't been > cached return "password mismatch". I though this might be a conflict with > nscd, but it happens whether nscd is running or not. So, for the time being, > we have had to disable the authentication cache.Authentication caching was more or less broken in all of alpha series, there are fixes for those in CVS.> I'd be very keen to hear from anyone who has any feedback on any of the > points above.Have you thought of clustered file system to go with your Dovecot hosts and CLARiiON device? -- Tomi Hakala
On Thursday 27 Oct 2005 17:42, Marcus Don wrote:> Hi > > We've been using Dovecot as our primary IMAP and POP3 server for around > 70,000 users for a couple of months now. On the whole, we've been very > pleased with the performance of Dovecot - and are extremely grateful to the > developers! However, we have had a few issues, some of which we have yet to > resolve. > > We chose Dovecot because it was the only IMAP/POP3 server we could find > that supported direct mysql authentication and both mboxes and maildirs. > With exim as our main SMTP server, we were able to convert the inboxes and > imap folders for all 70,000 users from mboxes to maildirs without any > downtime whatsoever! > > For those who are interested, we have a cluster of 5 web/imap servers > (running our own webmail software) and a further 5 pop servers all placed > behind 2 Zeus load balancers and running RHEL 4 (actually CentOS 4.1). > Storage is provided by an EMC Clariion device, which is mounted via NFS > onto all servers in the cluster over gigabit ethernet. > > 1. index files > > The main problem has always been the index files becoming corrupted. This > seems to have improved with the Alpha 4 release, but still happens for > several users each day. Perhaps it would be possible for Dovecot to just > delete index files and re-create them when they are corrupted, rather than > just erroring? Also, how do you turn off index files altogether? Even when > mail_never_cache_fields is set for all available fields, they still get > created.[snip]> > Thanks > > MarcusHi Marcus, Good to see another UK ISP using dovecot :) We use an LVS director with 4 pop/imap servers and a master-slave mysql backend for auth. The only problem we are still seeing that I am aware of is dovecot seems to sometimes rely on stale indexes rather than checking to see if they are correct first. This can cause clients to get message not found errors. I would debug properly but every time I get a moment free Manchester's IFL gets too hot and a server goes pop so I have to fix that instead. Regards Andrew -- Andrew Hutchings (A-Wing) - Linux Guru Netserve Consultants - http://www.domaincity.co.uk/ Linux CDs and DVDs - http://www.linuxiso.co.uk/ Random quote 206: "One World, one web, one program" - Microsoft Promo ad. "Ein Volk, Ein Reich, Ein Fuhrer" - Adolf Hitler -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://dovecot.org/pipermail/dovecot/attachments/20051027/dd8bfe59/attachment.pgp
On Thu, 2005-10-27 at 17:42 +0100, Marcus Don wrote:> 1. index files > > The main problem has always been the index files becoming corrupted. This > seems to have improved with the Alpha 4 release, but still happens for > several users each day.Reporting the exact errors here is useful. I do fix them once in a while :)> Perhaps it would be possible for Dovecot to just > delete index files and re-create them when they are corrupted, rather than > just erroring?It should do that. If it doesn't do it with some errors, please tell which errors and I'll fix it.> Also, how do you turn off index files altogether? Even when > mail_never_cache_fields is set for all available fields, they still get > created.That just means nothing is stored to dovecot.index.cache file. dovecot.index is still used.> Another issue relating to the index files is that, if a user is deleted and > then another user created with the same name, dovecot doesn't have > permissions to re-write the index files, because it has a different uid. So, > whenever a user is deleted, we have to delete the index files from every > machine in the cluster. We could store index files on the NFS device, but > this would affect performance (and might cause locking problems).In theory storing the indexes in NFS would be faster, if the mailbox was accessed from multiple computers.. But there are still some problems with it. Hmm. Wonder if it would be a good idea to delay the index updates and write them only when closing the mailbox. That way there would be less NFS traffic, but the indexes still would be updated globally, and if another computer accessed the mailbox later it would have already up-to-date indexes..> A better > solution would be to make the uid one of the variables available in > default_mail_env. By naming index files by uid rather than username, this > wouldn't be an issue.I guess.. Added %i to CVS.> 2. initgroups() > > We use an nss-mysql to store all non-administrative system users in a mysql > database. We often encounter problems with applications that use the > initgroups() function, since this returns all users and groups - which in > our case returns masses of data from mysql. When using mysql (or ldap etc) > for authentication, it would be useful if there were an option to prevent > additional system lookups. At present, we have to comment out the following > in /src/lib/restrict-access.c, or the server load goes through the roof: > > if (initgroups(env, gid) != 0) { > i_fatal("initgroups(%s, %s) failed: %m", env, dec2str(gid)); > }I think it's becuse of login processes, not because of imap/pop3 processes? The initgroups() is used only if userdb returns "system_user" parameter, which by default is only returned if you use passwd userdb. I'll change even that at some point so that auth process can just give a GID-list to be used. Hmm. And as for login process, it doesn't ever need more than one GID. I guess it shouldn't break anything if I just don't let it do the initgroups() call. Removed.> 3. base_dir permissions bug > > Since the alpha 4 release, it seems that the permissions dovecot > automatically sets for the base_dir are not sufficient to allow the > authentication user to access sockets, unless this user belongs to the same > group as the login user - which is contrary to the instructions in the > documentation. I'm pretty sure this is a bug, but perhaps someone could > confirm.You probably mean with mysql passdb/userdb? That's a bug..> 4. authentication caching > > Also since the alpha 4 release, we have found that, once the authentication > cache is full, all subsequent login attempts for users that haven't been > cached return "password mismatch". I though this might be a conflict with > nscd, but it happens whether nscd is running or not. So, for the time being, > we have had to disable the authentication cache.Looks like auth cache was pretty much completely broken in alpha4 for sql/ldap. Fixed. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://dovecot.org/pipermail/dovecot/attachments/20051028/1d123efa/attachment.pgp
Marcus Don wrote:> How confident are you of the CVS? Any idea when the next (perhaps beta?) > release will be?I run CVS code most of the time, but only with handful of users and with them it has been working OK. I wouldn't use CVS code in production without thorough stress testing in a lab first.>>Have you thought of clustered file system to go with your Dovecot >>hosts and CLARiiON device?> I'm not sure why this would benefit us at the moment. Before we switched to > maildirs, the bottleneck we were experiencing at busy times was the ethernet > speed, not the storage device speed.> Is there another reason you suggested this?No, I'm just not very fond of NFS myself but it seems that for you it is doing job pretty well. Your CLARiiON is a normal SAN device and you have some singe host providing NFS? It didn't quite open to me how your mail spool is set up, is it some meta LUN distributed over two disk arrays or do you just have one huge LUN and mirror it between two arrays? -- Tomi Hakala