Hi, About a week ago I upgraded our reasonably heavily loaded mail servers from a pretty recent courier version to dovecot-1.1rc10. IMAP performance on dovecot is amazing, however POP3 performance is worse than courier :-( I have uploaded some munin graphs taken today to http://linweb.atlas.pipex.net/dovecot/; the dovecot server is handling about 2000 pop logins/sec and 300 imap sessions (but these are negligible in terms of nfs ops). The courier server is handling a similar number of pop logins/sec but no imap (slightly less traffic overall but a similar number of logins). As you can see, dovecot seems to issue an awful lot more read requests than courier. The big spike at 10am is where I tried setting the :INDEX=MEMORY flag on dovecot as I assumed that's what courier is doing. I've also attached graphs of the relative network usage - as you can see, dovecot is using up to 100mbps whereas courier is using less than 10mbps; I believe this is due to the read. I initially assumed this high read load was due to courier building up its index files, but this has now been happening for about a week with little variation. The servers are both centos 5.0 (perhaps 5.1 actually, but with the latest centosplus kernel which fixes the nfs lookup bug in the standard kernels); and are not running any other nfs programs. The NFS server is a high performance NAS unit - no issues there. We're only using Maildirs; relevant config options are: mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes lock_method: dotlock maildir_copy_preserve_filename: yes Thanks, Mark
Sorr I can't give you an answer for your problem, but... On 6/30/2008 12:05 PM, Mark Zealey wrote:> About a week ago I upgraded our reasonably heavily loaded mail servers > from a pretty recent courier version to dovecot-1.1rc1o.1.1.1 was released on 6/22, so you're a bit behind the curve... its normally recommended to upgrade to a release version before reporting problems...> relevant config options are:Output of dovecot -n is preferred to selective snips from dovecot.conf... Also, dovecot is primarily an IMAP server, but someone else will have to tell you if the POP performance you are experiencing is normal or not... -- Best regards, Charles
On Mon, 2008-06-30 at 17:05 +0100, Mark Zealey wrote:> Hi, > > About a week ago I upgraded our reasonably heavily loaded mail servers > from a pretty recent courier version to dovecot-1.1rc10.rc11 had this fix which may be relevant: - dovecot-uidlist is now recreated if it results in file shrinking over 25%.> IMAP > performance on dovecot is amazing, however POP3 performance is worse > than courier :-( I have uploaded some munin graphs taken today to > http://linweb.atlas.pipex.net/dovecot/; the dovecot server is handling > about 2000 pop logins/sec and 300 imap sessions (but these are > negligible in terms of nfs ops). The courier server is handling a > similar number of pop logins/sec but no imap (slightly less traffic > overall but a similar number of logins). As you can see, dovecot seems > to issue an awful lot more read requests than courier.Dovecot's POP3 performance isn't that great, but it shouldn't be (much) worse than Courier's since they both use a nearly identical dovecot-uidlist/courierpop3dsizes files.> The big spike at > 10am is where I tried setting the :INDEX=MEMORY flag on dovecot as I > assumed that's what courier is doing.With v1.1 INDEX=MEMORY shouldn't matter that much anymore, since it should cache the virtual message size to dovecot-uidlist. 1) Do you have ,W=<size> in maildir filenames? Apparently not? 2) Do you have W=<size> added to dovecot-uidlist? 3) Are your POP3 users download + delete or do they leave mail on the server?> The servers are both centos 5.0 (perhaps 5.1 actually, but with the > latest centosplus kernel which fixes the nfs lookup bug in the standard > kernels); and are not running any other nfs programs. The NFS server is > a high performance NAS unit - no issues there. We're only using > Maildirs; relevant config options are: > > mmap_disable: yes > dotlock_use_excl: noYou should be able to use dotlock_use_excl=yes.> mail_nfs_storage: yes > mail_nfs_index: yes > lock_method: dotlockdotlock is the slowest locking method. Did you try how well fcntl would have worked? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20080701/6ddfc596/attachment-0002.bin>
> rc11 had this fix which may be relevant: > > - dovecot-uidlist is now recreated if it results in > file shrinking > over 25%.Not sure this fix is relevent; but I have upgraded to 1.1.1> With v1.1 INDEX=MEMORY shouldn't matter that much anymore, since it > should cache the virtual message size to dovecot-uidlist. > > 1) Do you have ,W=<size> in maildir filenames? Apparently not?No; yesterday though I did switch exim to delivering through lda - surprisingly though there was not much effect on the graphs.> 2) Do you have W=<size> added to dovecot-uidlist?I have found something here which I believe is a bug; will detail below> 3) Are your POP3 users download + delete or do they leave mail on the > server?A mixture; I believe at least 80% will download+delete; probably closer to 95% but we do have some very large mail stores - most of these would be accessed by a combination of pop and imap.> You should be able to use dotlock_use_excl=yes....> dotlock is the slowest locking method. Did you try how well > fcntl would > have worked?Not tried fcntl; I don't really trust nfs locking. I have enabled the excl in the config; again not much change. Now what I discovered in testing a particularly problematic user with an 800mb cur directory (under v1.1.1; I have never used v1.0 on this mbox but it was converted with your courier convert script); initially they had the following: drwx------ 2 exim exim 2048 Apr 11 2007 courierimapkeywords -rw-r--r-- 1 exim exim 47 Apr 22 15:56 courierimapsubscribed -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist drwx------ 2 exim exim 499712 Jul 2 10:35 cur -rw------- 1 exim exim 20912 Jul 2 10:51 dovecot.index -rw------- 1 exim exim 33056 Jun 17 17:34 dovecot.index.cache -rw------- 1 exim exim 31336 Jul 2 10:37 dovecot.index.log -rw------- 1 exim exim 158465 Jul 2 10:13 dovecot-uidlist drwx------ 2 exim exim 4096 Jul 2 10:29 new -rw------- 1 exim exim 23 Jun 24 22:49 subscriptions drwx------ 2 exim exim 2048 Jul 2 10:42 tmp As you can see, the cache file is very old, and the uidlist file is behind the last time that the logged in (which was 10:35). When I log in over pop and do a stat/list, it re-reads everything in cur/new but didn't update the dovecot-uidlist file or the index.cache file (the log file is updated though). Dovecot-uidlist file does not have the extra Wtags in it. I removed the dovecot.index.cache file and performed the same login/stat/list operation and logged out, and suddenly the dovecot-uidlist file was updated to the new v1.1 format, and the directory listing looks like: total 1068 drwx------ 2 exim exim 2048 Apr 11 2007 courierimapkeywords -rw-r--r-- 1 exim exim 47 Apr 22 15:56 courierimapsubscribed -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist drwx------ 2 exim exim 499712 Jul 2 10:35 cur -rw------- 1 exim exim 20912 Jul 2 10:53 dovecot.index -rw------- 1 exim exim 31428 Jul 2 10:53 dovecot.index.log -rw------- 1 exim exim 175508 Jul 2 10:53 dovecot-uidlist drwx------ 2 exim exim 4096 Jul 2 10:29 new -rw------- 1 exim exim 23 Jun 24 22:49 subscriptions drwx------ 2 exim exim 2048 Jul 2 10:42 tmp This looks like a bug to me; I will have a look in the source and see if I can find anything, but I thought I'd send this email as you'll be able to find the bug faster than I can. Thanks, Mark
In case anyone else is experiencing this issue, using the following in a postlogin script is a reasonable workaround: if [[ ! -e "$HOME/.lastlogin_imap" && "$HOME/Maildir/dovecot.index.cache" -ot "$HOME/Maildir/new" && "$HOME/Maildir/dovecot-uidlist" -ot "$HOME/Maildir/new" ]]; then rm -f "$HOME/Maildir/dovecot.index.cache" fi (whenever a user logs in via imap we touch .lastlogin_imap, so we only kill the cache file if the user is pop3-only and both the cache and uidlist are older than the new directory). Mark -- Mark Zealey -- Shared Hosting Team Leader Product Development * Webfusion 123-reg.co.uk, webfusion.co.uk, donhost.co.uk, supanames.co.uk This mail is subject to http://www.gxn.net/disclaimer> -----Original Message----- > From: dovecot-bounces+mark.zealey=webfusion.com at dovecot.org > [mailto:dovecot-bounces+mark.zealey=webfusion.com at dovecot.org] > On Behalf Of Mark Zealey > Sent: 02 July 2008 11:01 > To: Dovecot Mailing List > Subject: Re: [Dovecot] Poor pop3 over nfs performance > > > rc11 had this fix which may be relevant: > > > > - dovecot-uidlist is now recreated if it results in > > file shrinking > > over 25%. > > Not sure this fix is relevent; but I have upgraded to 1.1.1 > > > With v1.1 INDEX=MEMORY shouldn't matter that much anymore, since it > > should cache the virtual message size to dovecot-uidlist. > > > > 1) Do you have ,W=<size> in maildir filenames? Apparently not? > > No; yesterday though I did switch exim to delivering through lda - > surprisingly though there was not much effect on the graphs. > > > 2) Do you have W=<size> added to dovecot-uidlist? > > I have found something here which I believe is a bug; will > detail below > > > 3) Are your POP3 users download + delete or do they leave > mail on the > > server? > > A mixture; I believe at least 80% will download+delete; > probably closer > to 95% but we do have some very large mail stores - most of > these would > be accessed by a combination of pop and imap. > > > You should be able to use dotlock_use_excl=yes. > ... > > dotlock is the slowest locking method. Did you try how well > > fcntl would > > have worked? > > Not tried fcntl; I don't really trust nfs locking. I have enabled the > excl in the config; again not much change. > > Now what I discovered in testing a particularly problematic > user with an > 800mb cur directory (under v1.1.1; I have never used v1.0 on this mbox > but it was converted with your courier convert script); initially they > had the following: > > drwx------ 2 exim exim 2048 Apr 11 2007 courierimapkeywords > -rw-r--r-- 1 exim exim 47 Apr 22 15:56 courierimapsubscribed > -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb > -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist > drwx------ 2 exim exim 499712 Jul 2 10:35 cur > -rw------- 1 exim exim 20912 Jul 2 10:51 dovecot.index > -rw------- 1 exim exim 33056 Jun 17 17:34 dovecot.index.cache > -rw------- 1 exim exim 31336 Jul 2 10:37 dovecot.index.log > -rw------- 1 exim exim 158465 Jul 2 10:13 dovecot-uidlist > drwx------ 2 exim exim 4096 Jul 2 10:29 new > -rw------- 1 exim exim 23 Jun 24 22:49 subscriptions > drwx------ 2 exim exim 2048 Jul 2 10:42 tmp > > As you can see, the cache file is very old, and the uidlist file is > behind the last time that the logged in (which was 10:35). > When I log in > over pop and do a stat/list, it re-reads everything in cur/new but > didn't update the dovecot-uidlist file or the index.cache > file (the log > file is updated though). Dovecot-uidlist file does not have > the extra W> tags in it. I removed the dovecot.index.cache file and performed the > same login/stat/list operation and logged out, and suddenly the > dovecot-uidlist file was updated to the new v1.1 format, and the > directory listing looks like: > > total 1068 > drwx------ 2 exim exim 2048 Apr 11 2007 courierimapkeywords > -rw-r--r-- 1 exim exim 47 Apr 22 15:56 courierimapsubscribed > -rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb > -rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist > drwx------ 2 exim exim 499712 Jul 2 10:35 cur > -rw------- 1 exim exim 20912 Jul 2 10:53 dovecot.index > -rw------- 1 exim exim 31428 Jul 2 10:53 dovecot.index.log > -rw------- 1 exim exim 175508 Jul 2 10:53 dovecot-uidlist > drwx------ 2 exim exim 4096 Jul 2 10:29 new > -rw------- 1 exim exim 23 Jun 24 22:49 subscriptions > drwx------ 2 exim exim 2048 Jul 2 10:42 tmp > > This looks like a bug to me; I will have a look in the source > and see if > I can find anything, but I thought I'd send this email as > you'll be able > to find the bug faster than I can. > > Thanks, > > Mark >
Charles Marcus
2008-Jul-02 11:50 UTC
[Dovecot] OT - Legal disclaimers - WAS: Re: Poor pop3 over nfs performance
On 7/2/2008, Mark Zealey (Mark.Zealey at webfusion.com) wrote:> This mail is subject to http://www.gxn.net/disclaimerWow... good idea for disclaimers (I hate them)... I wonder if doing it this way fulfills the brain-dead laws requiring disclaimers some countries are enacting? -- Best regards, Charles