dovecot - Mar 2007 - index performance issues

Running 1.0.rc22

We have a single file server, running NFS (Single large sas disk) with 
about 80Gb of mail.
mount -o remount,rsize=2048,wsize=2044,actimeo=0,soft,rw 
xx.xx.xx.xx:/mailboxes /nfs1

There are 3 front end pop/imap servers using an LVS director.
Indexes are stored on each front end server.

-rw-------  1 admin 500   144 Mar 12 17:47 dovecot.index
-rw-------  1 admin 500 10272 Mar 12 17:47 dovecot.index.cache
-rw-------  1 admin 500   120 Mar 12 17:47 dovecot.index.log

Normally everything runs fine, there are about 1500 active pop 
accounts and 100 Imap.
Typically, 10 to 20 pop accounts are downloading at anyone time, and 
there may be 10 or 20 active Imap sessions.

If we find that the index partition has got to 100% full and corrupted 
the indexes then we'll delete the entire index dir and let dovecot 
rebuild all the indexes however, we end up with 100mbit / 10Mb/sec of 
network traffic between the NFS server and the pop/imap servers. The 
nfs server load climbs to 20+ as does the load on the front end mail 
servers.

It can take 4 or 5 hours for the indexes to rebuild during which time 
imap is slow but works and pop downloads all end up with dead 
processes that need clearing every few minutes.

admin     5869  0.0  0.1   4384  1060 ?        D    17:56   0:00  \_ 
pop3 [info at .co.uk 180.98]
admin     5877  0.0  0.1   4392  1112 ?        D    17:57   0:00  \_ 
pop3 [bro94 62.182]
admin     5879  0.0  0.1   4396  1052 ?        D    17:57   0:00  \_ 
pop3 [spamcatch at .com 81.163]
admin     5884  0.0  0.1   4392  1056 ?        D    17:57   0:00  \_ 
pop3 [ctaylor at .com 81.175]

Normally the nfs server delivers possibly 200Kbytes of traffic average 
spiking to 400Kbytes but the 900+Kbytes per second sustained rate for 
5 or 6 hours is killing the service.

Is there something we've got wrong here?

(I know a Scsi raid NFS would help as would Gbit networking but I 
can't see the setup we've got as needing that sort of hardware spec?)

Thanks

John

On 3/12/2007 Netserve (john at nsnoc.com) wrote:
> 
> If we find that the index partition has got to 100% full and 
> corrupted the indexes then we'll delete the entire index dir and let 
> dovecot rebuild all the indexes however,
<snip>
> Is there something we've got wrong here? 
I would say that considering it normal that the partition storing the 
indexes would routinely fill up and require deleting all indexes is wrong.

Increase the size of the partition storing the indexes to a sane size so 
that this doesn't happen.

Just my .02 clad coins worth

-- 

Best regards,

Charles

On 12.3.2007, at 20.01, Netserve wrote:
> If we find that the index partition has got to 100% full and  
> corrupted the indexes then we'll delete the entire index dir
The upcoming 1.0.rc27 won't break indexes if it runs out of disk  
space. But then again if it has already run out of disk space there,  
there's really nothing that frees more space.. Unless you keep  
running some find script to delete the oldest indexes.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070312/88fd3d09/attachment.bin>