dovecot - Mar 2008 - Effects of migration

So, to follow up to my previous thread, we just successfully migrated our 
NFS-based mail cluster from qmail pop, courier imap, and bincimap to 
dovecot 1.1rc1.

Overall the transition was very smooth, the only unexpected adjustment was 
having to implement ntpd on each box, rather than doing an hourly ntpdate 
against our local ntpd server, to prevent dovecot from crashing itself 
from too much drift.

The impact has been severe! Even with NFS-stored indexes, our netapp is 
seeing 1/6th of the NFS ops per second, and its CPU utilization is now at 
1/3rd previous levels.

The only user comment thus far was thanking us for bringing IMAP folders 
out from under INBOX. 

Dovecot is truly excellent. In my book, Timo joins Wietse Venema and Marc 
Martinec to form the backbone of the premiere open source mail solution.

For now, praise will have to suffice. I do, however, maintain a little IOU 
list that I intend to fulfill at some point in the future, and Timo is now 
high on the list.

Thanks again!
Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---

Words by Andy Dills [Wed, Mar 05, 2008 at 11:23:08AM
-0500]:
> 
> So, to follow up to my previous thread, we just successfully migrated our 
> NFS-based mail cluster from qmail pop, courier imap, and bincimap to 
> dovecot 1.1rc1.
> 
> Overall the transition was very smooth, the only unexpected adjustment was 
> having to implement ntpd on each box, rather than doing an hourly ntpdate 
> against our local ntpd server, to prevent dovecot from crashing itself 
> from too much drift.
> 
> The impact has been severe! Even with NFS-stored indexes, our netapp is 
> seeing 1/6th of the NFS ops per second, and its CPU utilization is now at 
> 1/3rd previous levels.
> 
Yeah. We went from a solution like yours to a based on dovecot imapd and
we reduced the servers in 1/3 and the server load to half. Pretty
impressive :)

-- 
Jose Celestino
----------------------------------------------------------------
http://www.msversus.org/     ; http://techp.org/petition/show/1
http://www.vinc17.org/noswpat.en.html
----------------------------------------------------------------
"If you would have your slaves remain docile, teach them hymns."
    -- Ed Weathers ("The Empty Box")

On Wed, 2008-03-05 at 11:23 -0500, Andy Dills wrote:
> The impact has been severe! Even with NFS-stored indexes, our netapp is 
> seeing 1/6th of the NFS ops per second, and its CPU utilization is now at 
> 1/3rd previous levels.
Have you thought about enabling Squat indexes? I'd like to know how much
it would affect I/O and CPU usage in larger installations. CPU grows
(maybe a lot) but searches should be faster and use very little I/O as a
result.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080306/e44b7c43/attachment-0002.bin>

On Thu, 6 Mar 2008, Timo Sirainen wrote:
> On Wed, 2008-03-05 at 11:23 -0500, Andy Dills wrote:
> 
> > The impact has been severe! Even with NFS-stored indexes, our netapp
is
> > seeing 1/6th of the NFS ops per second, and its CPU utilization is now
at
> > 1/3rd previous levels.
> 
> Have you thought about enabling Squat indexes? I'd like to know how
much
> it would affect I/O and CPU usage in larger installations. CPU grows
> (maybe a lot) but searches should be faster and use very little I/O as a
> result.
By CPU, do you mean local server (nfs client) CPU or netapp CPU (nfs 
server)?

I'm guessing the former...for what it's worth, for those who have yet to
have the pleasure of using a Netapp, the CPU utilization is basically your 
ultimate barometer of utilization with netapps. As long as you don't start 
hitting 90% CPU consistently, they will provide better throughput than 
local SCSI disks. The CPU tops out well before the I/O, a nice change of 
pace. 

Local CPU is of little concern typically, as mail serving (filtering is 
handled elsewhere) is almost purely I/O.

However, I'm not sure how much value I would place on optimizing searches 
at this point...do users really do much of that? It seems to be 
potentially valuable yet underutilized.

Do you have some links so I can educate myself more about squat indexes?

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---