dovecot - Aug 2021 - [OT] Re: What kind of search response time are you setting with solr full text search?

> 
> This is offtopic for this list, but I will try to help you.? If I am
> unsuccessful, you should raise the issue on the solr-users mailing
> list.
Thanks so much. Greatly appreciated.
> How much of the total server memory of 4GB did you give to Solr for
> its heap?? Is there other software running on that server besides
> Solr?
I'm not sure and didn't know I could assign memory to solr. This is the 
first time I've used solr; I'm totally inexperienced with it. I've 
attached a screenshot that hopefully helps answer your question.

This machine is dedicated to solr and runs no other software as a daemon 
not related to the OS.

> 
> What's the total size of all the Solr indexes on the Solr server?
du -shc on /var/solr/data reports 781 M of total data.

du -sch on the core I'm using is only 116 M. As of now, I have 15K 
documents with about 2K deleted in the core.

> 
> Can you get the screenshot mentioned at the following URL, put it on a
> file-sharing site, and give me the URL?
> 
>
https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue
Attached.

> 
> (disclaimer: I wrote that Solr wiki page)
> 
> You should read the entire page, the link above is to the section
> describing useful screenshots)
OK, I will.
> 
> General note:? A Solr search that takes 3 seconds (let alone 15) would
> have me concerned.? If the system is sized appropriately, I would
> expect a search even on a massive index to complete in less than a
> second.
> 
> I happen to be using Solr for dovecot myself.? If I search my index
> for "the" which is very common in English text, the query takes
19
> milliseconds, and that is searching on five fields, as well as doing a
> facet on the user field.? My Solr index has 150048 messages (122K of
> those are in my personal mailbox) and takes up 628 megabytes of disk
> space.? The total size of the email that is indexed is 7 gigabytes.
Interesting and good to know.
> 
> |<str name="parsedquery_toString">+(cc:the | from:the |
to:the |
> body:the | subject:the)</str> |
> 
> My index is using the stopword filter but the list of stopwords is 
> empty.
> 
> The following response may interest you:
> 
> https://apaste.info/hm42
> 
> This is a search for "a" which I had run several times, so Solr
was
> serving it from its cache, and this time it only took 6 milliseconds.?
> It also shows what a facet can do.? The longest time I got for the
"a"
> search was 15 milliseconds, before the query was in the cache.
I think they queries themselves are returning very quickly, at least 
they were when I did a query on a single inbox. That's why I'm wondering
if doing a search across 40 different inboxes via roundcube might be the 
issue. I'm thinking that each time a new mailbox is selected, it slows 
things down. But I have no idea how the IMAP search is performed across 
multiple inboxes so that's just a wild guess.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-08-25 at 1.00.46 PM.png
Type: image/png
Size: 42649 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210825/056688d6/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-08-25 at 1.07.38 PM.png
Type: image/png
Size: 1462070 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210825/056688d6/attachment-0003.png>

>> 
>>
https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue
> 
> Attached.
> 
I uploaded the wrong screenshot. Brain was on a badly flawed autopilot 
system.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2021-08-25 at 1.15.05 PM.png
Type: image/png
Size: 412311 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210825/888facc6/attachment-0001.png>

>> 
>> This is a search for "a" which I had run several times, so
Solr was
>> serving it from its cache, and this time it only took 6 milliseconds.?
>> It also shows what a facet can do.? The longest time I got for the
"a"
>> search was 15 milliseconds, before the query was in the cache.
> 
> I think they queries themselves are returning very quickly, at least
> they were when I did a query on a single inbox. That's why I'm
> wondering if doing a search across 40 different inboxes via roundcube
> might be the issue. I'm thinking that each time a new mailbox is
> selected, it slows things down. But I have no idea how the IMAP search
> is performed across multiple inboxes so that's just a wild guess.
OK, I take this back. I did an imap search via telnet and solr reports 
the search takes about 3 to 4 seconds. Here's the output:

a search text "maynez"
* SEARCH 5 6 7 32 61 64 69 70 117 118 119 120 121 122 123 124 126 127 
129 165 197 202 203 204 205 206 207 216 231 259 451 452 453 454 455 456 
482 730 731 810 811 812 813 814 815 816 817 818 819 820 829 830 831 832 
852 853 854 855 867 868 869 870 871 872 873 874 875 886 887 888 889 891 
904 908 909 910 911 912 913 920 924 925 926 927 928 931 936 938 940 941 
944 946 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 
964 965 966 969 970 971 972 973 974 975 976 977 986 987 988 989 993 1012 
1013 1014 1015 1016 1017 1019 1020 1021 1022 1023 1024 1025 1026 1027 
1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 
1044 1047 1048 1069 1091 1112 1113 1114 1121 1126 1127 1128 1129 1130 
1131 1132 1133 1134 1135 1136 1144 1145 1146 1147 1148 1150 1159 1160 
1161 1162 1175 1176 1177 1178 1179 1180 1181 1186 1187 1188 1189 1190 
1192 1193 1197 1198 1202 1204 1205 1208 1214 1215 1216 1217 1218 1228 
1229 1231 1232 1234 1236 1237 1243 1244 1245 1246 1247 1248 1249 1250 
1252 1256 1257 1297 1298 1299 1300 1301 1302 1303 1304 1307 1308 1316 
1317 1319 1320 1327 1328 1331 1332 1335 1336 1348 1349 1352 1355 1356 
1358 1359 1361 1417 1418 1419 1420 1421 1423 1424 1443 1444 1445 1446 
1447 1453 1460 1461 1462 1463 1464 1500 1501 1502 1503 1504 1505 1507 
1508 1509 1510 1513 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 
1532 1533 1534 1535 1539 1541 1542 1543 1544 1594 1596 1597 1598 1599 
1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 
1620 1621 1622 1623 1628 1629 1631 1632 1633 1634 1635 1636 1637 1638 
1639 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1660 1661 1663 
1664 1665 1666 1674 1675 1676 1677 1679 1680 1688 1691 1693 1694 1695 
1696
a OK Search completed (3.029 + 0.001 + 0.228 secs).