dovecot - Apr 2019 - FTS delays

On 2 Apr 2019, at 6.38, Joan Moreau via dovecot <dovecot at dovecot.org>
wrote:
> 
> Further on this topic:
> 
> 
> 
> When choosing any headers in the search box, dovecot core calls the plugin
TWICE (and returns the results quickly, but not immediatly after getting the IDs
from the plugins)
> 
> When choosing the BODY search, dovecot core calls the plugin ONCE (and
never returns) (whereas the plugins returns properly the IDs)
> 
If we simplify this, do you mean this calls it once and is fast:

doveadm search -u user at domain mailbox inbox body helloworld

But this calls twice and is slow:

doveadm search -u user at domain mailbox inbox text helloworld

And what about searching e.g. subject? :

doveadm search -u user at domain mailbox inbox subject helloworld

And does the slowness depend on whether there were any matches or not?
> This is based on GIT version. (previous versions were working properly)
Previous versions were fast? Do you mean v2.3.5?

Example from real life 
>From Roubdcube, I serach "milan" in full message (body &
headers)
Logs : 

Apr 3 10:24:01 gjserver dovecot[29778]:
imap(jom at grosjo.net)<30311><4pACp52FfCF/AAAB>: Query : (
bcc:milan OR
body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan OR uid:milan )
Apr 3 10:24:01 gjserver dovecot[29778]:
imap(jom at grosjo.net)<30311><4pACp52FfCF/AAAB>: Query: 81 results
in 2 ms


81 results is correct 

but Roundcube times out 

from command line, I do : 

doveadm search -u jom at grosjo.net mailbox inbox text milan 

output 

doveadm(jom at grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR
cc:inbox OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox
OR uid:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR from:milan
OR message-id:milan OR subject:milan OR to:milan OR uid:milan )
doveadm(jom at grosjo.net): Info: Query: 1 results in 1 ms
d82b4b0f550d38593644000095331209 847
d82b4b0f550d38593644000095331209 1569
d82b4b0f550d38593644000095331209 2260
d82b4b0f550d38593644000095331209 2575
d82b4b0f550d38593644000095331209 2811
d82b4b0f550d38593644000095331209 2885
d82b4b0f550d38593644000095331209 3038
d82b4b0f550d38593644000095331209 3121
d82b4b0f550d38593644000095331209 3170 

1 - The query is wrong 

2 - teh last line "d8...209 3170" gets repeated for ages 

On 2019-04-02 16:30, Timo Sirainen wrote:
> On 2 Apr 2019, at 6.38, Joan Moreau via dovecot <dovecot at
dovecot.org> wrote:
> 
>> Further on this topic:
>> 
>> When choosing any headers in the search box, dovecot core calls the
plugin TWICE (and returns the results quickly, but not immediatly after getting
the IDs from the plugins)
>> 
>> When choosing the BODY search, dovecot core calls the plugin ONCE (and
never returns) (whereas the plugins returns properly the IDs)
> 
> If we simplify this, do you mean this calls it once and is fast:
> 
> doveadm search -u user at domain mailbox inbox body helloworld
> 
> But this calls twice and is slow:
> 
> doveadm search -u user at domain mailbox inbox text helloworld
> 
> And what about searching e.g. subject? :
> 
> doveadm search -u user at domain mailbox inbox subject helloworld
> 
> And does the slowness depend on whether there were any matches or not?
> 
>> This is based on GIT version. (previous versions were working properly)
> 
> Previous versions were fast? Do you mean v2.3.5?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190403/aadeceb0/attachment.html>

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot <dovecot at dovecot.org>
wrote:
> doveadm search -u jom at grosjo.net mailbox inbox text milan
Did that search over my list mail and got 83 results, not able to duplicate your
issue.

What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.


-- 
There is a tragic flaw in our precious Constitution, and I don't know
what can be done to fix it. This is it: Only nut cases want to be
president.

On 3 Apr 2019, at 20.30, Joan Moreau via dovecot <dovecot at dovecot.org>
wrote:
> doveadm search -u jom at grosjo.net mailbox inbox text milan
> output
> 
> doveadm(jom at grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR
cc:inbox OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR
uid:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR
message-id:milan OR subject:milan OR to:milan OR uid:milan )
> 
> 1 - The query is wrong
That's because fts_backend_xapian_lookup() isn't anywhere close to being
correct. Try to copy the logic based on solr_add_definite_query_args().