Stan Hoeppner
2010-Feb-23 14:19 UTC
[Dovecot] body search very slow since upgrade from 1.0.15 to 1.2.10
Did you mislead me Timo? You said search in 1.1+ is faster than 1.0. I'm seeing approximately 20x *slower* search times in 1.2.10. Via Thunderbird, a full body search of my 11,000+ message IMAP folder hosted by 1.0.15 used to take less than 10 seconds. Since upgrading to 1.2.10 the search is taking over 3 minutes, the imap process servicing the client pegging one CPU at 100% for the duration of the search. Here's the mbox file: -rw------- 1 stan stan 50M Feb 23 07:41 1-Spam-l Doing the same search via grep results in a time of: real 0m0.361s user 0m0.204s sys 0m0.140s grep takes less than half a second, yet dovecot 1.2.10 takes 3 minutes for the same search? Actually grep is searching the entire file including headers, so it's doing more work than the imap process. Something is *SERIOUSLY* wrong here. Did I somehow nerfed something in dovecot.conf that's causing this? I deleted all the index files as per the upgrade instructions. However, checking the directories shows they're apparently rebuilt: [08:09:15][root at greer]/home/stan/mail/.imap/1-Spam-l$ la total 28M drwx------ 2 stan stan 4.0K Feb 23 07:41 . drwx------ 35 stan stan 4.0K Feb 15 11:14 .. -rw------- 1 stan stan 358K Feb 23 07:18 dovecot.index -rw------- 1 stan stan 27M Feb 23 07:41 dovecot.index.cache -rw------- 1 stan stan 616 Feb 23 07:41 dovecot.index.log What do I need to check/change/do to get these search times back down to reasonable/respectable levels, i.e. around 10s? Thanks. -- Stan
Timo Sirainen
2010-Feb-23 18:33 UTC
[Dovecot] body search very slow since upgrade from 1.0.15 to 1.2.10
On 23.2.2010, at 16.19, Stan Hoeppner wrote:> Did you mislead me Timo? You said search in 1.1+ is faster than 1.0. I'm > seeing approximately 20x *slower* search times in 1.2.10. > > Via Thunderbird, a full body search of my 11,000+ message IMAP folder hosted > by 1.0.15 used to take less than 10 seconds. Since upgrading to 1.2.10 the > search is taking over 3 minutes, the imap process servicing the client > pegging one CPU at 100% for the duration of the search. Here's the mbox file:Well, v1.1+ does case-insensitive searches by converting all mails to UTF-8 first. But it really shouldn't make it that much slower. What OS and CPU is this with? You could also try if it happens to work faster by compiling Dovecot without iconv support. I guess there could also be some kind of a bug that you're triggering.
Stan Hoeppner
2010-Feb-24 07:21 UTC
[Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Stan Hoeppner put forth on 2/24/2010 12:17 AM:> Timo Sirainen put forth on 2/23/2010 11:06 PM:>> No, body searches aren't indexed (without fts plugins).I wish you'd have mentioned the Squat plugin earlier Timo. I feel I've made a fool of myself. I just setup squat and after a long first search on that 50MB, 11K+ messages mbox file, every search after the first has taken less than 5 seconds, most only 2 seconds. It's performing exactly as I'd like. It's amazing. I didn't ever setup any fts plugins on 1.0.15. I'm left wondering how body searches were so fast if I didn't have an fts plugin configured. Anyway, setting up Squat on 1.2.10 sure solved the long search time problem. :) Thanks for the help Timo! -- Stan