dovecot - Apr 2007 - keeping indexes in tmpfs

While doing some testing with converting accounts while simulating 
incoming mail load (no other pop/imap processes going but 4 processes 
converting users), we found that we were maxing out the local disk in 
the server with the index activity. To find out that it was the index 
activity, I mounted a tmpfs for dovecot to keep indexes on, and the 
system load dropped from 70 to 3 :)

Anyway, Timo and others, do you have any thoughts about keeping indexes 
on a tmpfs partition? I realize the obvious issues with this: running 
out of space, and the fact that the partition is lost on reboot. 
However, we plan to run a cluster of these servers anyway, where we will 
keep domains/accounts going to a particular server but fail over to one 
of the others in case of an outage. Recreating the indexes seems pretty 
trivial when writing to tmpfs anyway, plus we could copy the indexes to 
disk (restore after reboot) in the case of planned outages... unplanned 
outages result in another server rebuilding them all anyway.


So, I'm looking for non-obvious issues we may have with this approach. I 
need to do more testing to see how much memory we would actually need to 
feasibly keep the indexes there, but otherwise is there a good reason 
not to do this?

On Mon, 2007-04-16 at 09:37 -0400, Justin McAleer wrote:
> While doing some testing with converting accounts while simulating 
> incoming mail load (no other pop/imap processes going but 4 processes 
> converting users), we found that we were maxing out the local disk in 
> the server with the index activity. To find out that it was the index 
> activity, I mounted a tmpfs for dovecot to keep indexes on, and the 
> system load dropped from 70 to 3 :)
How exactly were you converting the users? I guess if the system is
building new index files for tons of users that could take a while.
fsync_disable=yes could help a lot with reducing the disk writes, and
maybe also mmap_disable=yes.

For Dovecot v2.0 I'm planning on reducing at least dovecot.index.log
file sizes. I think currently it uses way too much space when building
the initial indexes.
> Anyway, Timo and others, do you have any thoughts about keeping indexes 
> on a tmpfs partition? I realize the obvious issues with this: running 
> out of space, and the fact that the partition is lost on reboot. 
I think memory would be better used for keeping mailbox data cached
that's actually useful at the time.

Also if you've POP3 users who keep messages in the server and
dovecot.index.cache file is lost, all the messages are read to calculate
the messages' virtual sizes when the user logs in the next time. That
could be expensive.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070416/fbb9b1bc/attachment.bin>

Timo Sirainen wrote:
> On Mon, 2007-04-16 at 09:37 -0400, Justin McAleer wrote:
>   
>> While doing some testing with converting accounts while simulating 
>> incoming mail load (no other pop/imap processes going but 4 processes 
>> converting users), we found that we were maxing out the local disk in 
>> the server with the index activity. To find out that it was the index 
>> activity, I mounted a tmpfs for dovecot to keep indexes on, and the 
>> system load dropped from 70 to 3 :)
>>     
>
> How exactly were you converting the users? I guess if the system is
> building new index files for tons of users that could take a while.
> fsync_disable=yes could help a lot with reducing the disk writes, and
> maybe also mmap_disable=yes.
>
>   
Ok, I'll give more detail about the setup. I plan to use the convert 
plugin to migrate from CommuniGate to dovecot for our ~160,000 accounts. 
We will probably migrate one domain at a time, letting users basically 
migrate their mail at first login. But, we will also go through all the 
domain's accounts logging in to pop3 just to ensure everything gets moved.

So, for testing, I've copied a few of our domains' mail spools over to 
the test server and started a script that forks and goes through all 
their accounts and simply logs in to pop3 and waits for a message list 
to come back. We have the maildirs being stored on an NFS mount though, 
and indexes on localdisk, so the only localdisk activity was logging and 
indexes. I'll try setting both of those options and see how much 
difference it makes, although I'm not sure that losing mail upon server 
reset is acceptible for us (disabling fsync).
> For Dovecot v2.0 I'm planning on reducing at least dovecot.index.log
> file sizes. I think currently it uses way too much space when building
> the initial indexes.
>
>   
>> Anyway, Timo and others, do you have any thoughts about keeping indexes
>> on a tmpfs partition? I realize the obvious issues with this: running 
>> out of space, and the fact that the partition is lost on reboot. 
>>     
>
> I think memory would be better used for keeping mailbox data cached
> that's actually useful at the time.
>   
Perhaps, but one way or another, we apparently cannot use a simple 
localdisk for indexes. So, the options are either memory or some sort of 
raid setup. Since we use maildirs, isn't the disk cache aspect less of a 
concern?
> Also if you've POP3 users who keep messages in the server and
> dovecot.index.cache file is lost, all the messages are read to calculate
> the messages' virtual sizes when the user logs in the next time. That
> could be expensive.
>
>