dovecot - Feb 2004 - More Benchmarks, and a question about indexing

Hi,

We are thinking of converting to Dovecot to lighten our I/O load. I have done
some performace tests comparing UW-IMAP to Dovecot when using mbox format if
anyone is interested. They primarily examine the I/O improvements that can be
gained from using Dovecot over UW when staying with mbox format:

http://whizzo.uoregon.edu/public/src/mailperf/results.html

I had too much cached for this set of tests, and I am going to re-run some
with the recommended cache settings...so stay tuned.

Anyway, we are stuck with the mbox format for the forseeable future, and I
noticed how hard hitting Dovecot is on initial access to a large mailbox, so we
need a way to pre-create some of those indexes. I saw in the docs that such an
indexer has been planned, but does anyone have a hack for doing it now?

-- 
Anthony Kay                        Phone: (541) 346-1719
Student Consultant                 Key Signature: 7195 40B9 FC4F 6F43 C04A  
University of Oregon Computing Center             B4D9 BAF1 C76E AB6E 33E5

"Don't wish your life away."

   - Mom's best advice

Anthony Wendell Kay wrote:
> Hi,
> 
> We are thinking of converting to Dovecot to lighten our I/O load. I have
done
> some performace tests comparing UW-IMAP to Dovecot when using mbox format
if
> anyone is interested. They primarily examine the I/O improvements that can
be
> gained from using Dovecot over UW when staying with mbox format:
Perhaps I just missed something - but what was used to benchmark? IMHO? 
I'm curious as I wouldn't mind trying to benchmark (in general) my 
Maildir setup. Granted, I don't have an mbox comoparison anymore for the 
same machine, but it would be an interesting one none the less.
> Anyway, we are stuck with the mbox format for the forseeable future, and I
> noticed how hard hitting Dovecot is on initial access to a large mailbox,
so we
> need a way to pre-create some of those indexes. I saw in the docs that such
an
> indexer has been planned, but does anyone have a hack for doing it now?
IMHO, it's not a difficult transition as long as you have a good sized 
window to make the swap (depending on number of users and speed of the 
hardware), and your MDA is capable of switching easily (Procmail is). I 
found/wrote a couple of scripts to facilitate the transition, which were 
recently sent to this list. Perhaps there are other (political?) reasons 
for sticking w/ mbox, however.

Myth dispelling time: For those who are concerned about inode usage with 
Maildir, my 110GB available RAID is 21% used on disk space with mail, 
but only 4% used for inodes. It seems to me I'll run out of space long 
before I run out of inodes with my user's present usage. This is for 150 
users, approx 60% of which use IMAP and 1/2 of them again store 
significant amounts of mail on the server. Some of my largest users have 
upwards of 30,000 messages (packrat managers).

-Rick
-- 
Rick Johnson, RHCE #807302311706007 - rjohnson at medata.com
Linux/Network Administrator - Medata, Inc. (from home)
PGP Public Key: https://mail.medata.com/pgp/rjohnson.asc

On Tue, 2004-02-17 at 02:47, Anthony Wendell Kay wrote:
> We are thinking of converting to Dovecot to lighten our I/O load. I have
done
> some performace tests comparing UW-IMAP to Dovecot when using mbox format
if
> anyone is interested. They primarily examine the I/O improvements that can
be
> gained from using Dovecot over UW when staying with mbox format:
> 
> http://whizzo.uoregon.edu/public/src/mailperf/results.html
Good testing. I was going to do something similiar myself, but only for
the next release which will have rewritten index and mbox code. I know
the current mbox code is really slow and buggy in some situations.

I've however heard of getting much larger performance benefits when
switching from UW-IMAP. I think they had mostly Outlook/OE clients.
Drops from average 20MB/s disk I/O to few kB/s.
> Anyway, we are stuck with the mbox format for the forseeable future, and I
> noticed how hard hitting Dovecot is on initial access to a large mailbox,
so we
> need a way to pre-create some of those indexes. I saw in the docs that such
an
> indexer has been planned, but does anyone have a hack for doing it now?
This, and POP access will work better in next version too. Currently
Dovecot goes and stores all kinds of cached data to index files even if
client never uses them. This will change so that data is stored only
when client actually asks for it which I think makes opening large
mailboxes almost as fast as UW-IMAP (small hit for writing 200-500kB
indexes per 10000 mails).

I'll also create Dovecot LDA which is able to update the indexes
directly while storing the mail, that should help some too.

Anyway, back to coding. I hope to get it usable soon and committed to
CVS. Before that I'm pretty unresponsive..

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20040218/1b8e43a5/attachment-0001.bin>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Timo Sirainen wrote:

|>Anyway, we are stuck with the mbox format for the forseeable future, and I
|>noticed how hard hitting Dovecot is on initial access to a large
mailbox, so we
|>need a way to pre-create some of those indexes. I saw in the docs that
such an
|>indexer has been planned, but does anyone have a hack for doing it now?
|
|
| This, and POP access will work better in next version too. Currently
| Dovecot goes and stores all kinds of cached data to index files even if
| client never uses them. This will change so that data is stored only
| when client actually asks for it which I think makes opening large
| mailboxes almost as fast as UW-IMAP (small hit for writing 200-500kB
| indexes per 10000 mails).
|
| I'll also create Dovecot LDA which is able to update the indexes
| directly while storing the mail, that should help some too.
|
| Anyway, back to coding. I hope to get it usable soon and committed to
| CVS. Before that I'm pretty unresponsive..

Cool!

~ - Jonas

- --
* Jonas Smedegaard - idealist og Internet-arkitekt
* Tlf.: +45 40843136  Website: http://dr.jones.dk/

~ - Enden er n?r: http://www.shibumi.org/eoti.htm
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAM5Pqn7DbMsAkQLgRAnDUAJ9DgE+mJr3Az2qNbGsVqovb6ClTZwCfbNyg
BWbW3gDv6lMe6zFhZZTtnls=iEU2
-----END PGP SIGNATURE-----