dovecot - Aug 2021 - [OT] Re: What kind of search response time are you setting with solr full text search?

On 8/24/2021 7:19 PM, Steve Dondley wrote:
>> THE PROBLEM: When I do a full text search through all my inbox and all
>> subfolders on a single word, search results are returned in about 10
>> to 15 seconds. This is better than the 40 seconds or so I'm getting
>> when I turn off the fts and fts_solr plugins but still a little
>> disappointing.
>
> I did some experimenting. I noticed that if the word I'm searching on 
> is fairly rare, results will pop up quickly, like in around 3 to 5 
> seconds. Words that don't exist at all in any email returns nothing 
> almost instantly.
>
> But words that appear in several hundred emails are the ones that are 
> take a much longer time. 
This is offtopic for this list, but I will try to help you.? If I am 
unsuccessful, you should raise the issue on the solr-users mailing list.

How much of the total server memory of 4GB did you give to Solr for its 
heap?? Is there other software running on that server besides Solr?

What's the total size of all the Solr indexes on the Solr server?

Can you get the screenshot mentioned at the following URL, put it on a 
file-sharing site, and give me the URL?

https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue

(disclaimer: I wrote that Solr wiki page)

You should read the entire page, the link above is to the section 
describing useful screenshots)

General note:? A Solr search that takes 3 seconds (let alone 15) would 
have me concerned.? If the system is sized appropriately, I would expect 
a search even on a massive index to complete in less than a second.

I happen to be using Solr for dovecot myself.? If I search my index for 
"the" which is very common in English text, the query takes 19 
milliseconds, and that is searching on five fields, as well as doing a 
facet on the user field.? My Solr index has 150048 messages (122K of 
those are in my personal mailbox) and takes up 628 megabytes of disk 
space.? The total size of the email that is indexed is 7 gigabytes.

|<str name="parsedquery_toString">+(cc:the | from:the | to:the |
body:the | subject:the)</str> |

My index is using the stopword filter but the list of stopwords is empty.

The following response may interest you:

https://apaste.info/hm42

This is a search for "a" which I had run several times, so Solr was 
serving it from its cache, and this time it only took 6 milliseconds.? 
It also shows what a facet can do.? The longest time I got for the "a"
search was 15 milliseconds, before the query was in the cache.

Thanks,
Shawn

> 
> This is offtopic for this list, but I will try to help you.? If I am
> unsuccessful, you should raise the issue on the solr-users mailing
> list.
Thanks so much. Greatly appreciated.
> How much of the total server memory of 4GB did you give to Solr for
> its heap?? Is there other software running on that server besides
> Solr?
I'm not sure and didn't know I could assign memory to solr. This is the 
first time I've used solr; I'm totally inexperienced with it. I've 
attached a screenshot that hopefully helps answer your question.

This machine is dedicated to solr and runs no other software as a daemon 
not related to the OS.

> 
> What's the total size of all the Solr indexes on the Solr server?
du -shc on /var/solr/data reports 781 M of total data.

du -sch on the core I'm using is only 116 M. As of now, I have 15K 
documents with about 2K deleted in the core.

> 
> Can you get the screenshot mentioned at the following URL, put it on a
> file-sharing site, and give me the URL?
> 
>
https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue
Attached.

> 
> (disclaimer: I wrote that Solr wiki page)
> 
> You should read the entire page, the link above is to the section
> describing useful screenshots)
OK, I will.
> 
> General note:? A Solr search that takes 3 seconds (let alone 15) would
> have me concerned.? If the system is sized appropriately, I would
> expect a search even on a massive index to complete in less than a
> second.
> 
> I happen to be using Solr for dovecot myself.? If I search my index
> for "the" which is very common in English text, the query takes
19
> milliseconds, and that is searching on five fields, as well as doing a
> facet on the user field.? My Solr index has 150048 messages (122K of
> those are in my personal mailbox) and takes up 628 megabytes of disk
> space.? The total size of the email that is indexed is 7 gigabytes.
Interesting and good to know.
> 
> |<str name="parsedquery_toString">+(cc:the | from:the |
to:the |
> body:the | subject:the)</str> |
> 
> My index is using the stopword filter but the list of stopwords is 
> empty.
> 
> The following response may interest you:
> 
> https://apaste.info/hm42
> 
> This is a search for "a" which I had run several times, so Solr
was
> serving it from its cache, and this time it only took 6 milliseconds.?
> It also shows what a facet can do.? The longest time I got for the
"a"
> search was 15 milliseconds, before the query was in the cache.
I think they queries themselves are returning very quickly, at least 
they were when I did a query on a single inbox. That's why I'm wondering
if doing a search across 40 different inboxes via roundcube might be the 
issue. I'm thinking that each time a new mailbox is selected, it slows 
things down. But I have no idea how the IMAP search is performed across 
multiple inboxes so that's just a wild guess.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-08-25 at 1.00.46 PM.png
Type: image/png
Size: 42649 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210825/056688d6/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-08-25 at 1.07.38 PM.png
Type: image/png
Size: 1462070 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210825/056688d6/attachment-0003.png>

One other data point from my experimenting that might shed some light on 
the problem:

If I limit a search to a single folder instead of across all folders, it 
still takes 5 or 6 seconds for the results to appear. So that kind of 
destroys my theory that the problem might be caused by having too many 
inbox folders.