Hello All, I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection. Talking in the #dovecot channel I was gave a scenario of a system supporting ~40k users with 4 servers just fine. I wonder how well dovecot would scale if we increase this number of users by some order of magnitude like, say, 4M users. It is well known that preforking is a good pratice if you want to achieve a higher performance. When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't. Do you have any plans to support preforking in the near future? Best regards, giorgenes
Words by Giorgenes Gelatti [Thu, Aug 14, 2008 at 03:38:50PM -0300]:> Hello All, > > I've been studying dovecot for replacing my company's current system > and I got a little worried about an aspect of the dovecot's design. > I was surprised that dovecot doesn't use prefork for its mail > processes, forking a new processes for each new client connection. > > Talking in the #dovecot channel I was gave a scenario of a system > supporting ~40k users with 4 servers just fine. > I wonder how well dovecot would scale if we increase this number of > users by some order of magnitude like, say, 4M users. >Well, we have 8 servers for that amount of users.> It is well known that preforking is a good pratice if you want to > achieve a higher performance.Some say it's premature optimization.> When I was asked about it I readily answered: "of course it does". For > my surprise later, i doesn't. > > Do you have any plans to support preforking in the near future? >-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc ---------------------------------------------------------------- "One man?s theology is another man?s belly laugh." -- Robert A. Heinlein
On Aug 14, 2008, at 2:38 PM, Giorgenes Gelatti wrote:> I've been studying dovecot for replacing my company's current system > and I got a little worried about an aspect of the dovecot's design. > I was surprised that dovecot doesn't use prefork for its mail > processes, forking a new processes for each new client connection.Login processes are preforked and they can also be configured to keep reusing existing processes. But yes, post-login imap/pop3 processes aren't preforked because it would require a larger redesign of the master process. Which is actually what I have been planning on doing for v2.0 for a long time now. But there are even some theoretical problems with preforking. For example the most secure way to set up your users is to use a different UNIX UID for each user. So for preforking that means your preforked processes must run as root until they receive the information about which UID they need to run as. And the code running as root should be minimized.. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 194 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20080814/b5b46e87/attachment-0002.bin>
2008/8/14 Timo Sirainen <tss at iki.fi>:> But there are even some theoretical problems with preforking. For example > the most secure way to set up your users is to use a different UNIX UID for > each user. So for preforking that means your preforked processes must run as > root until they receive the information about which UID they need to run as. > And the code running as root should be minimized..True, but it's a common scenario to have thousands of users, in which case they usually have all the same UID. []'s
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, Aug 14, 2008 at 03:38:50PM -0300, Giorgenes Gelatti wrote:> Hello All,[...]> It is well known that preforking is a good pratice if you want to > achieve a higher performance. > When I was asked about it I readily answered: "of course it does". For > my surprise later, i doesn't.With fork latencies in the range of 500 to 1500 microseconds (on Pentium 900 MHz-class hardware!) on most modern kernels[1] I wonder whether this "good practice" isn't on the verge of voodoo ;-) (Of course, in a http server, where you might expect thousands of connects per second, this is another story -- which is mitigated by HTTP 1.1, when properly streaming several requests per connection). - ---------------- [1] <http://bulk.fefe.de/scalability/>, search "The fork benchmark" Regards - -- tom?s -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIpR1XBcgs9XrR2kYRAqPdAJ0dbp+fUW0MpWdNvXa3SUvXP3v3eQCcCsTS hFbhMpoG+OjI4i+za6xNn+4=SRgx -----END PGP SIGNATURE-----
Hi Timo, On 14 Aug 2008, at 17:04, Timo Sirainen wrote:> On Aug 14, 2008, at 2:38 PM, Giorgenes Gelatti wrote: > >> I've been studying dovecot for replacing my company's current system >> and I got a little worried about an aspect of the dovecot's design. >> I was surprised that dovecot doesn't use prefork for its mail >> processes, forking a new processes for each new client connection. > > Login processes are preforked and they can also be configured to > keep reusing existing processes. But yes, post-login imap/pop3 > processes aren't preforked because it would require a larger > redesign of the master process. Which is actually what I have been > planning on doing for v2.0 for a long time now.Do you have pointer(s) to some mails / wiki pages talking about your ideas of implementation of pre-forked process pool? Regards, Sebastien Tandel