Hi, Are there any big Dovecot sites (say 50,000-100,000) users on this list? What sort of hardware do you use. What sort of problems do you see? Many thanks, Jonathan.
* On 21/09/06 18:14 +0100, Jonathan wrote:
| Hi,
|
| Are there any big Dovecot sites (say 50,000-100,000) users on this
| list?
Hmm, I doubt but there could be. My biggest deployment has 8800 users.
A few of those could be inactive as I need to do some housekeeping.
| What sort of hardware do you use.
HP DL380, single 3.0GHz CPU, 2GB RAM, SCSI disks.
| What sort of problems do you see?
Other than dovecot dying unexpectedly when compiled with kqueue support
on FreeBSD 6.1, nothing else. I had to disable Kqueue support.
Some tuning is required though. So far I have only bumped up the value
of "login_max_processes_count" to 200. Since then, it's been
running
non-stop for 10 days without any hiccup.
-Wash
http://www.netmeister.org/news/learn2quote.html
DISCLAIMER: See http://www.wananchi.com/bms/terms.php
--
+======================================================================+
|\ _,,,---,,_ | Odhiambo Washington <wash at wananchi.com>
Zzz /,`.-'`' -. ;-;;,_ | Wananchi Online Ltd. www.wananchi.com
|,4- ) )-,_. ,\ ( `'-'| Tel: +254 20 313985-9 +254 20 313922
'---''(_/--' `-'\_) | GSM: +254 722 743223 +254 733
744121
+======================================================================+
If A equals success, then the formula is _A = _X + _Y + _Z. _X is work. _Y
is play. _Z is keep your mouth shut.
-- Albert Einstein
Jonathan ?????:> Hi, > > Are there any big Dovecot sites (say 50,000-100,000) users on this > list? > > What sort of hardware do you use. > > What sort of problems do you see? > > Many thanks, > Jonathan.Small install, dovecot 1.0rc7 on FreeBSD 6.1/kqueue, sometimes imap process eats up to 90% CPU, ktrace shows about 20000 calls of gettimeofday() per second. I mentioned about this problem already in this list. -- ? ?????????, ?????? ????? ??? "???????" : ?????????? ??, WEB-?????????? http://www.elantech.ru +7 (495) 589 68 81 +7 (926) 575 22 11
On Thu, Sep 21, 2006 at 06:14:14PM +0100, Jonathan wrote:> Are there any big Dovecot sites (say 50,000-100,000) users on this > list?Yes. Actually, more users than that.> What sort of hardware do you use.2 x NetApp FAS2070c clustered filer heads w/NFS (fast fibrechannel disk shelves) IBM x336 SMTP/POP3/IMAP front-ends, 4GB RAM, 2x3GHz CPU. Load-balancing across front-ends via Cisco 6509 SLB. Gigabit Ethernet, jumbo frames.> What sort of problems do you see?You need a stable NFS client. We're using Linux 2.6.16 with NFS client patches from Trond Myklebust. nfs mount options: 10.x.x.x:/vol/mailstore /var/mailstore nfs rw,tcp,nfsvers=3,rsize=8192,wsize=8192,nosuid,nodev,soft,intr,noac,actimeo=0 0 0 dovecot.conf - mmap_disable=yes - lock_method=fnctl It's extremely fast & stable (server loads rarely above 1.0), serving a six-figure userbase their spam, their mailbombs, and just occasionally, some genuine mail :) Other major platform components: OpenLDAP 2.3, Postfix 2.3. The main bottleneck is Postfix spooling before delivery, but I have to generate synthetic mailbombs to make it seriously blink (and I'm still in the "Dovecot probably needs a LMTP speaker someday" camp). We can eliminate even that with /var/spool/postfix on a tmpfs, but I'm unwilling to go that far. Note: without Trond M's NFS client patchset we see comedy VFS out-of-sync errors after a few days uptime, resulting in mistaken deadlocking of dovecot indices (usually only one user, but always a high-volume user) that needs a node reboot to fix, and some intervention with the "lock status / break" command on the OnTAP command line. With the patches it's been rock solid. If only I could make anti-spam services run so fast! Timo, could you design a mail hygiene system next? Thanks :) all the best JG -- Josh "Koshua" Goodall "as modern as tomorrow afternoon" joshua at roughtrade.net - FW109
On Thu, September 21, 2006 1:14 pm, Jonathan <dovecot at redigloo.org> said:> Are there any big Dovecot sites (say 50,000-100,000) users on this > list?Yes, Webmail.us. Many multiples of that number :)> What sort of hardware do you use.80 backend IMAP servers. 5 front-end IMAP-proxy servers. All running Dovecot. The boxes are single AMD CPU, 1GB RAM, and 4 SATA drives. DRBD used to replicate data between servers.> What sort of problems do you see?Dovecot doesn't need much CPU, but it needs a lot of disk. So give it lots of spindles, either in one big box or many small boxes like we do. Right now the only problem we have is the Dovecot proxy software sometimes disconnects imap sessions that it should not disconnect. Since Timo is very busy, we are working on a patch for this. http://www.dovecot.org/list/dovecot/2006-August/015401.html Bill
Bill Boebel wrote:> 80 backend IMAP servers. 5 front-end IMAP-proxy servers. All > running Dovecot. The boxes are single AMD CPU, 1GB RAM, and 4 SATA > drives. DRBD used to replicate data between servers.I have a similar setup (though much, much, much smaller, appr. 35000 mailboxes). I have them split over two backend servers, and two frontend servers. Anti-spam and anti-virus is done on the backends (because we do that per mailbox). -How many mailboxes do you run on one backend? -Do you have any anti-spam/virus running on them or is that done earlier? -Have you ever used perdition and if so, how does dovecot as a proxy compare to that? -Do you mind me asking these questions? Robin
On Fri, September 22, 2006 5:05 am, Robin Elfrink <robin at 15augustus.nl> said:> -How many mailboxes do you run on one backend?The limit on the backend is disk. We're comfortable at about 10,000 mailboxes per 7200 RPM SATA drive. The more disks you put on each backend server, the more mailboxes each server can handle.> -Do you have any anti-spam/virus running on them or is that done earlier?Yes. Currently 17 servers doing just filtering. These run Postfix, SpamAssassin, Policyd, ClamAV.> -Have you ever used perdition and if so, how does dovecot as a proxy > compare to that?Yes. Perdition requires a process per connection though, so it does not scale well. With Dovecot's proxy (and also the nginx proxy, http://sysoev.ru/en/) individual processes can handle multiple connections, and takes advantage of epoll for network events which is ideal for handling large numbers of idle IMAP connections. With Perdition we'd run at 2000 connections per server, with Dovecot we haven't it a peak yet, but expect to peak at around 15,000 - 25,000 connections per proxy server. Obviously the bigger the box, the more connections it can handle.> -Do you mind me asking these questions?Nope. Ask away. If I can't answer it I'll tell you. Bill
On Thu, 2006-09-21 at 21:16 -0400, Bill Boebel wrote:> > Are there any big Dovecot sites (say 50,000-100,000) users on this > > list? > Yes, Webmail.us. Many multiples of that number :)Many means in the 10^6 order of magnitude? More?> 80 backend IMAP servers. 5 front-end IMAP-proxy servers. All running Dovecot. The boxes are single AMD CPU, 1GB RAM, and 4 SATA drives. DRBD used to replicate data between servers.AFAIK DRBD can replicate data at the block device layer. So I imagine only half of the backend servers are actually active (with a twin HA passive host, or you have 80x2 actual servers) and frontends are redirecting users to the correct backend IMAP server. Is this correct? ciao, Luca