-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
this may be an obvious logical problem I'm not aware of, which cannot be
solved
any more efficiently... but maybe it's just a bug or there is potential for
optimisation in Dovecot (or Thunderbird?).
When searching for multiple terms at once ("any of") with
Thunderbird/Dovecot
(using FTS Squat indexes), it takes much longer (not double the time, or a bit
more than that, but really much, much longer) than when I search for only one
term.
Here are some figures (the IMAP commands are shown as issued by Thunderbird and
recorded by Dovecot rawlog).
While searching, I can see the 'imap' process using 100% CPU time.
Searching for "xyz" in Archive and subfolders
4 secs
32 select "Archive"
33 uid SEARCH UNDELETED BODY "xyz"
34 select "Archive.2004"
35 UID fetch 1:* (FLAGS)
36 uid SEARCH UNDELETED BODY "xyz"
37 select "Archive.2004.2005"
38 UID fetch 1:* (FLAGS)
39 uid SEARCH UNDELETED BODY "xyz"
40 select "Archive.2004.2005.2006"
41 UID fetch 1:* (FLAGS)
42 uid SEARCH UNDELETED BODY "xyz"
43 select "Archive.2004.2005.2006.2007"
44 UID fetch 1:* (FLAGS)
45 uid SEARCH UNDELETED BODY "xyz"
46 select "Archive.2004.2005.2006.2007.2008"
47 UID fetch 1:* (FLAGS)
48 uid SEARCH UNDELETED BODY "xyz"
49 IDLE
Searching for "xyz" and "abc" (any of) in Archive and
subfolders
123 secs
50 select "Archive"
51 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
52 select "Archive.2004"
53 UID fetch 1:* (FLAGS)
54 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
55 select "Archive.2004.2005"
56 UID fetch 1:* (FLAGS)
57 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
58 select "Archive.2004.2005.2006"
59 UID fetch 1:* (FLAGS)
60 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
61 select "Archive.2004.2005.2006.2007"
62 UID fetch 1:* (FLAGS)
63 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
64 select "Archive.2004.2005.2006.2007.2008"
65 UID fetch 1:* (FLAGS)
66 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
67 IDLE
Searching for "xyz", "abc" and "def" in Archive
and subfolders
173 secs
68 select "Archive"
69 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
70 select "Archive.2004"
71 UID fetch 1:* (FLAGS)
72 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
73 select "Archive.2004.2005"
74 UID fetch 1:* (FLAGS)
75 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
76 select "Archive.2004.2005.2006"
77 UID fetch 1:* (FLAGS)
78 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
79 select "INBOX"
80 UID fetch 1:* (FLAGS)
81 check
82 select "Archive.2004.2005.2006.2007"
83 UID fetch 1:* (FLAGS)
84 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
85 select "Archive.2004.2005.2006.2007.2008"
86 UID fetch 1:* (FLAGS)
87 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY
"def")
88 IDLE
Searching for "xyz", "abc", "def" and
"ghi" in Archive and subfolders
217 secs
89 select "Archive"
90 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
91 select "Archive.2004"
92 UID fetch 1:* (FLAGS)
93 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
94 select "Archive.2004.2005"
95 UID fetch 1:* (FLAGS)
96 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
97 select "Archive.2004.2005.2006"
98 UID fetch 1:* (FLAGS)
99 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
100 select "Archive.2004.2005.2006.2007"
101 UID fetch 1:* (FLAGS)
102 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
103 select "Archive.2004.2005.2006.2007.2008"
104 UID fetch 1:* (FLAGS)
105 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc")
BODY "def") BODY "ghi")
106 IDLE
Patrick.
- --
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/
Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc
Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIooxq7yMg/OiDoAURAhb9AJ4jOaK+eHXcSwJTAER8xEKIlJJt0gCgtZan
wJm05V0F03gj+6YxbNOtCew=jID/
-----END PGP SIGNATURE-----
On Aug 13, 2008, at 3:25 AM, Patrick Nagel wrote:> this may be an obvious logical problem I'm not aware of, which > cannot be solved > any more efficiently... but maybe it's just a bug or there is > potential for > optimisation in Dovecot (or Thunderbird?). > > When searching for multiple terms at once ("any of") with > Thunderbird/Dovecot > (using FTS Squat indexes), it takes much longer (not double the > time, or a bit > more than that, but really much, much longer) than when I search for > only one term.Yea, OR isn't supported by FTS plugin yet.. I had hoped no-one would notice it ;) I guess I should do something about it. Requires a lot more complexity to the code though.. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 194 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20080813/3a80ed68/attachment-0002.bin>
Timo Sirainen wrote:> Yea, ...I've been meaning to tell you that should be "Yeah" for an informal version of "Yes", otherwise it's a very archaic form of "Yes" or "Indeed" as in "Yea, though I walk in the valley of the shadow of death"! I think that's also true of American/Aussie etc. as well ... Best Wishes, Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin at reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 13 Aug 2008 17:18:37 -0400, Timo Sirainen <tss at iki.fi> wrote:> On Aug 13, 2008, at 3:25 AM, Patrick Nagel wrote: > >> When searching for multiple terms at once ("any of") with >> Thunderbird/Dovecot >> (using FTS Squat indexes), it takes much longer (not double the >> time, or a bit >> more than that, but really much, much longer) than when I search for >> only one term. > > Yea, OR isn't supported by FTS plugin yet.. I had hoped no-one would > notice it ;) I guess I should do something about it.Heh, I see ;) As it is, the FTS plugin is quite useless to us though :( most of the time people are doing a (real world) search, they don't remember the exact word they're looking for. For example they only remember that the mail they're looking for contained "English" and "automobile" - or was it "Englisch" and "Automobil", because the mail was in German? So to get a list of all possible mails, they want to make use of the OR connection of search terms...> Requires a lot > more complexity to the code though..Couldn't Dovecot just search the index with each single word (as it does now without OR), and then merge the results, filtering out double UIDs? This is probably not the most efficient way to solve this, but it would be definitely *much* faster than not using the indexes at all - and doesn't sound too complex to me. Are there any other not-yet-implemented SEARCH features that don't work with the FTS plugin? Thanks, Patrick. - -- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779 PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: http://getfiregpg.org iEYEARECAAYFAkijga4ACgkQ7yMg/OiDoAUnFwCfXDUDHR+UdlMKkqwn2/nkB61I X3MAn1nK8KVW8RbtzQ6qQGipGb3uMXRa =I8Eh -----END PGP SIGNATURE-----