I thought I'd try benchmarking with dbmail (v2.2.4) to see how much
slower a SQL backend could actually be. Skip to bottom for the
conclusions.
Originally I ran the tests with the databases being in XFS
filesystem. MySQL's performance was horrible. It went 3-7x faster
with ext3.
MySQL 5.0.30 backend (innodb):
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0
Logi Sele Appe
100% 100% 100%
5%
1 291 303
So that's 10 messages/sec saved. Now how about with 5 concurrent
clients?
Logi Sele Appe
100% 100% 100%
5%
5 1259 1332
Pretty well. Then something more generic:
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30% 5%
1 37 36 75 73 110 34 24 73 78
Without Fetc (header/metadata fetching):
Logi List Stat Sele Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 50% 100% 100% 100%
30% 5%
1 94 94 199 283 102 85 198 210
PostgreSQL 8.1.5 backend:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0
Logi Sele Appe
100% 100% 100%
5%
1 267 277
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0
Logi Sele Appe
100% 100% 100%
5%
5 1094 1144
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30% 5%
9 29 40 74 72 99 22 12 64 71
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0
Logi List Stat Sele Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 50% 100% 100% 100%
30% 5%
35 105 95 200 277 54 70 165 175
The last two tests gave "Unexpected tagged reply:" errors that I
didn't get with MySQL. So apparently there's some PostgreSQL-specific
bug in dbmail.
The same values with Dovecot 1.0 + maildir:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0
Logi Sele Appe
100% 100% 100%
5%
1 346 364
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0
Logi Sele Appe
100% 100% 100%
5%
5 1408 1470
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30% 5%
1 130 133 259 258 382 123 127 257 271
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0
Logi List Stat Sele Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 50% 100% 100% 100%
30% 5%
1 155 163 339 478 169 175 338 354
So, what are the conclusions?
- In raw append speed dbmail is almost as fast as maildir.
- In raw read/write speed maildir is about 1,6 times faster
- When adding metadata fetches Dovecot is 4 times faster than
dbmail. This is most likely because dbmail doesn't have a cache
equivalent to dovecot.index.cache so it has to do the fetches the
slow way.
I would have liked to also run the generic tests with more than 1
client, but then I start hitting dbmail bugs and the test stops
(reported already to their bugtracker).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070512/ca097f37/attachment.bin>
Could someone please update the dbmail wikipedia page with this info ;) http://en.wikipedia.org/wiki/DBMail_IMAP_and_POP3_server I suppose what would be nicer would be to extend the dovecot entry and reference that from the dbmail entry ;) On Sat, May 12, 2007 at 03:16:05AM +0300, Timo Sirainen wrote:> I thought I'd try benchmarking with dbmail (v2.2.4) to see how much > slower a SQL backend could actually be. Skip to bottom for the > conclusions. > > Originally I ran the tests with the databases being in XFS > filesystem. MySQL's performance was horrible. It went 3-7x faster > with ext3. > > MySQL 5.0.30 backend (innodb): > > ./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 > Logi Sele Appe > 100% 100% 100% > 5% > 1 291 303 > > So that's 10 messages/sec saved. Now how about with 5 concurrent > clients? > > Logi Sele Appe > 100% 100% 100% > 5% > 5 1259 1332 > > Pretty well. Then something more generic: > > ./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% > 30% 5% > 1 37 36 75 73 110 34 24 73 78 > > Without Fetc (header/metadata fetching): > Logi List Stat Sele Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 50% 100% 100% 100% > 30% 5% > 1 94 94 199 283 102 85 198 210 > > PostgreSQL 8.1.5 backend: > > ./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 > Logi Sele Appe > 100% 100% 100% > 5% > 1 267 277 > > ./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 > Logi Sele Appe > 100% 100% 100% > 5% > 5 1094 1144 > > ./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% > 30% 5% > 9 29 40 74 72 99 22 12 64 71 > > ./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 > Logi List Stat Sele Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 50% 100% 100% 100% > 30% 5% > 35 105 95 200 277 54 70 165 175 > > The last two tests gave "Unexpected tagged reply:" errors that I > didn't get with MySQL. So apparently there's some PostgreSQL-specific > bug in dbmail. > > The same values with Dovecot 1.0 + maildir: > > ./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 > Logi Sele Appe > 100% 100% 100% > 5% > 1 346 364 > > ./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 > Logi Sele Appe > 100% 100% 100% > 5% > 5 1408 1470 > > ./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% > 30% 5% > 1 130 133 259 258 382 123 127 257 271 > > ./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 > Logi List Stat Sele Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 50% 100% 100% 100% > 30% 5% > 1 155 163 339 478 169 175 338 354 > > So, what are the conclusions? > > - In raw append speed dbmail is almost as fast as maildir. > - In raw read/write speed maildir is about 1,6 times faster > - When adding metadata fetches Dovecot is 4 times faster than > dbmail. This is most likely because dbmail doesn't have a cache > equivalent to dovecot.index.cache so it has to do the fetches the > slow way. > > I would have liked to also run the generic tests with more than 1 > client, but then I start hitting dbmail bugs and the test stops > (reported already to their bugtracker).-- -------------------------------------------------------------------------- Troy Benjegerdes 'da hozer' hozer at hozed.org Somone asked me why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: "Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz
Timo Sirainen wrote:> I thought I'd try benchmarking with dbmail (v2.2.4) to see how much > slower a SQL backend could actually be. Skip to bottom for the > conclusions. > > Originally I ran the tests with the databases being in XFS filesystem. > MySQL's performance was horrible. It went 3-7x faster with ext3. > >Timo, what was the hardware you used for this test?> > - In raw append speed dbmail is almost as fast as maildir. > - In raw read/write speed maildir is about 1,6 times faster > - When adding metadata fetches Dovecot is 4 times faster than dbmail. > This is most likely because dbmail doesn't have a cache equivalent to > dovecot.index.cache so it has to do the fetches the slow way. > > I would have liked to also run the generic tests with more than 1 > client, but then I start hitting dbmail bugs and the test stops > (reported already to their bugtracker).
On Sat, 2007-05-12 at 03:16 +0300, Timo Sirainen wrote:> - In raw append speed dbmail is almost as fast as maildir. > - In raw read/write speed maildir is about 1,6 times faster > - When adding metadata fetches Dovecot is 4 times faster than > dbmail. This is most likely because dbmail doesn't have a cache > equivalent to dovecot.index.cache so it has to do the fetches the > slow way.I think these are wrong actually. In those benchmarks the server was mainly just waiting for new APPEND commands, so the results were somewhat similar. Now lets try with 100 clients to see how it goes: dbmail SVN 2_2_branch + MySQL: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 1642 Hmm. Maybe that was too much for it. Lets try with 30: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=30 Logi Appe 100% 100% 5% 30 2620 So the performance got better with less simultaneous clients. Dovecot CVS HEAD + maildir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 4720 The speed dropped from 200 msgs/sec to 100 msgs/sec. The larger the maildir, the slower it becomes. Lets try with 30 clients with Dovecot too: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=30 Logi Appe 100% 100% 5% 30 3960 msgs/sec drops, as expected. Now, lets see how much fsync is slowing it down: Dovecot CVS HEAD + maildir + fsync_disable=yes: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 5951 The speed drops from 587/sec -> 113/sec. Lets try Dovecot's high performance Cyrus-like cydir format then: Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 35217 Well, isn't that something. Almost 10 times faster than maildir. The speed only drops from 1240/sec to 1150/sec. fsync_disable=yes gives +1000 messages, so not that much. Since the speed was quite steady 11-12/connection, lets try with some more connections. How about 200 clients: Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=200 Logi Appe 100% 100% 5% 200 1398 200/200 0 2268 200/200 0 2145 200/200 0 1965 200/200 0 1448 200/200 0 1650 200/200 0 1216 200/200 0 1193 200/200 0 1112 200/200 ... 0 708 200/200 0 812 200/200 0 726 200/200 0 780 200/200 0 727 200/200 Totals: Logi Appe 100% 100% 5% 200 32844 This didn't go so well anymore. It started nicely with the expected 2200/sec, but dropped rapidly. Might be because of lock contention, or because of ext3, or even because of imaptest itself. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070516/f6bef240/attachment.bin>