THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...) On 2018-12-05 12:20, Aki Tuomi wrote:> It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated. > > Also Solr does not support prefix/substring search unless you configure solr to support it. > > Aki > > On 5.12.2018 11.50, Joan Moreau wrote: > > Well, "break-imap-search" option prevent dovecot from starting. > > I used solr on one server since yesterday. > > First feedback is that > > * BODY search is not working > > * FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search > > On 2018-12-05 07:55, Aki Tuomi wrote: > > Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing. > > Updated the wiki a bit. > > Aki > > On 4.12.2018 20.50, Joan Moreau via dovecot wrote: > > also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki > > and not sure then how to enable full text search in BODY > > On 2018-12-04 19:40, Joan Moreau via dovecot wrote: > > In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : > > sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance > > before updating the schema.xml . > > Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users > > Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki) > > On 2018-12-04 19:01, Joan Moreau via dovecot wrote: > > Hi > > I am giving Solr another try. > > Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) > > Maybe some things have moved ? > > Thank you so muchLinks: ------ [1] http://(solr -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/64dc8d61/attachment.html>
Squat has not been maintained since 2.2.something, and shall not be maintained. Sorry. If you want, you can become maintainer for squat plugin. Aki On 5.12.2018 13.57, Joan Moreau wrote:> > THen the Squat shall be maintained until the SOlr plugin is upgraded, > as Squat does resolve the problem (fts, partial search, etc...) > > > ? > > > On 2018-12-05 12:20, Aki Tuomi wrote: > >> It seems we forgot to document that "break-imap-search" was dropped >> in v2.3. That has now been updated. >> >> Also Solr does not support prefix/substring search unless you >> configure solr to support it. >> >> Aki >> >> On 5.12.2018 11.50, Joan Moreau wrote: >>> >>> Well, "break-imap-search" option prevent dovecot from starting. >>> >>> >>> I used solr on one server since yesterday. >>> >>> >>> First feedback is that >>> >>> * BODY search is not working >>> >>> * FTS is not really working in headers : For instance, if you search >>> "kliinik" but you type "kliini" (missing one k), the search does not >>> return anything , whereas it shall return more results (as the >>> keyword is smaller) that the intended search >>> >>> >>> ? >>> >>> >>> On 2018-12-05 07:55, Aki Tuomi wrote: >>> >>> Full text search gets enabled automatically, although your >>> probably want to set fts_enforced=yes to make sure it's used for >>> all kinds of things automatically. break-imap-search is >>> supported,? but does nothing. >>> >>> Updated the wiki a bit. >>> >>> Aki >>> >>> On 4.12.2018 20.50, Joan Moreau via dovecot wrote: >>> >>> also, the value "break-imap-search" is not supported in >>> fts_solr parameter, contrary to the wiki >>> >>> and not sure then how to enable full text search in BODY >>> >>> ? >>> >>> >>> On 2018-12-04 19:40, Joan Moreau via dovecot wrote: >>> >>> In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr >>> ), it would nice to stipulate to the reader? to type the >>> command : >>> >>> sudo -u solr /opt/solr/bin/solr create -c dovecot # to >>> create the dovecot instance >>> >>> before updating the schema.xml . >>> >>> Also,? schema.xml is in >>> /opt/solr/server/solr/dovecot/conf for archlinux users >>> >>> Additionaly, the url is http://(solr_ >>> <http://(solr>server):8983/solr/dovecot/ (error in wiki) >>> >>> >>> ? >>> >>> >>> On 2018-12-04 19:01, Joan Moreau via dovecot wrote: >>> >>> Hi >>> >>> I am giving Solr another try. >>> >>> Using Solr 7.5 (from Archlinux AUR), I do not see >>> anymore the"solr/conf/schema.xml" neither the >>> "managed-schema" folder. (as >>> per?https://wiki.dovecot.org/Plugins/FTS/Solr)? >>> >>> Maybe some things have moved ? >>> >>> Thank you so much >>> >>> >>>-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/424244e6/attachment-0001.html>
Additionally, I get "Invalid search response" on my Android , using dovecot-solr, whereas squat works fine Why making squat obsolete ? It is simple and straightforward On December 5, 2018 12:57:22 PM Joan Moreau via dovecot <dovecot at dovecot.org> wrote:> THen the Squat shall be maintained until the SOlr plugin is upgraded, as > Squat does resolve the problem (fts, partial search, etc...) > > > > On 2018-12-05 12:20, Aki Tuomi wrote: >> It seems we forgot to document that "break-imap-search" was dropped in >> v2.3. That has now been updated. >> Also Solr does not support prefix/substring search unless you configure >> solr to support it. >> Aki >> On 5.12.2018 11.50, Joan Moreau wrote: >>> Well, "break-imap-search" option prevent dovecot from starting. >>> >>> I used solr on one server since yesterday. >>> >>> First feedback is that >>> * BODY search is not working >>> * FTS is not really working in headers : For instance, if you search >>> "kliinik" but you type "kliini" (missing one k), the search does not return >>> anything , whereas it shall return more results (as the keyword is smaller) >>> that the intended search >>> >>> >>> >>> On 2018-12-05 07:55, Aki Tuomi wrote: >>> Full text search gets enabled automatically, although your probably want to >>> set fts_enforced=yes to make sure it's used for all kinds of things >>> automatically. break-imap-search is supported, but does nothing. >>> Updated the wiki a bit. >>> Aki >>> On 4.12.2018 20.50, Joan Moreau via dovecot wrote: >>> also, the value "break-imap-search" is not supported in fts_solr parameter, >>> contrary to the wiki >>> and not sure then how to enable full text search in BODY >>> >>> >>> On 2018-12-04 19:40, Joan Moreau via dovecot wrote: >>> In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice >>> to stipulate to the reader to type the command : >>> sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot >>> instance >>> before updating the schema.xml . >>> Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users >>> Additionaly, the url is http://(solr_server):8983/solr/dovecot/ (error in wiki) >>> >>> >>> >>> On 2018-12-04 19:01, Joan Moreau via dovecot wrote: >>> Hi >>> I am giving Solr another try. >>> Using Solr 7.5 (from Archlinux AUR), I do not see anymore >>> the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per >>> https://wiki.dovecot.org/Plugins/FTS/Solr) >>> Maybe some things have moved ? >>> Thank you so much-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/628d74ec/attachment.html>
On Wed Dec 05 2018 07:35:39 GMT-0500 (Eastern Standard Time), Joan Moreau via dovecot <dovecot at dovecot.org> wrote:> Why making squat obsolete ? It is simple and straightforwardBecause no one has stepped up to maintain it?
After some testsing, I managed to get proper functionning
- The schema.xml is attached below (quite different from the one
provided on teh wiki) (in bold the core differences) (NGramFilterFactory
is the class that replace the fts_squat "partial=3 full=15",
everything
else is just a big hammer to smash a tiny fly)
- One need to remove the "managed-schema" file in the
{prefix}/server/solr/dovecot/conf.
- One need to remove everything under {prefix}/server/solr/dovecot/data/
- The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see
diff)
- my dovecot.conf is :
fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_decoder = decode2text
fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/
--- schema.xml
<?xml version="1.0" encoding="UTF-8"?>
<schema name="dovecot" version="2.0">
<uniqueKey>id</uniqueKey>
<FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" />
<FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" />
<FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" />
<fieldType name="text" class="solr.TextField"
autoGeneratePhraseQueries="true"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory"
words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1"
splitOnNumerics="1" catenateWords="1"
catenateNumbers="1"
catenateAll="1"/>
<filter class="solr.FlattenGraphFilterFactory"/> <!--
required on index
analyzers after graph filters -->
<filter class="solr.LowerCaseFilterFactory"/>
<FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3"
MAXGRAMSIZE="15"
/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymGraphFilterFactory"
expand="true"
ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/> <!--
required on index
analyzers after graph filters -->
<filter class="solr.StopFilterFactory"
words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1"
splitOnNumerics="1" catenateWords="1"
catenateNumbers="1"
catenateAll="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3"
MAXGRAMSIZE="15"
/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="_version_" type="long"
indexed="true" stored="true"/>
<field name="bcc" type="text" indexed="true"
stored="false"/>
<field name="body" type="text" indexed="true"
stored="false"/>
<field name="box" type="string" indexed="true"
required="true"
stored="true"/>
<field name="cc" type="text" indexed="true"
stored="false"/>
<field name="from" type="text" indexed="true"
stored="false"/>
<field name="hdr" type="text" indexed="true"
stored="false"/>
<field name="id" type="string" indexed="true"
required="true"
stored="true"/>
<field name="subject" type="text"
indexed="true" stored="false"/>
<field name="to" type="text" indexed="true"
stored="false"/>
<field name="uid" type="long" indexed="true"
required="true"
stored="true"/>
<field name="user" type="string" indexed="true"
required="true"
stored="true"/>
</schema>
------ diff solrconfig
--- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505
+0000
+++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000
@@ -1128,6 +1128,7 @@
See http://wiki.apache.org/solr/GuessingFieldTypes
-->
+ <schemaFactory
class="ClassicIndexSchemaFactory"></schemaFactory>
<updateProcessor class="solr.UUIDUpdateProcessorFactory"
name="uuid"/>
<updateProcessor
class="solr.RemoveBlankFieldUpdateProcessorFactory"
name="remove-blank"/>
<updateProcessor
class="solr.FieldNameMutatingUpdateProcessorFactory"
name="field-name-mutating">
@@ -1158,43 +1159,8 @@
<str>yyyy-MM-dd</str>
</arr>
</updateProcessor>
- <updateProcessor
class="solr.AddSchemaFieldsUpdateProcessorFactory"
name="add-schema-fields">
- <lst name="typeMapping">
- <str name="valueClass">java.lang.String</str>
- <str name="fieldType">text_general</str>
- <lst name="copyField">
- <str name="dest">*_str</str>
- <int name="maxChars">256</int>
- </lst>
- <!-- Use as default mapping instead of defaultFieldType -->
- <bool name="default">true</bool>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Boolean</str>
- <str name="fieldType">booleans</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.util.Date</str>
- <str name="fieldType">pdates</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Long</str>
- <str name="valueClass">java.lang.Integer</str>
- <str name="fieldType">plongs</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Number</str>
- <str name="fieldType">pdoubles</str>
- </lst>
- </updateProcessor>
<!-- The update.autoCreateFields property can be turned to false to
disable schemaless mode -->
- <updateRequestProcessorChain
name="add-unknown-fields-to-the-schema"
default="${update.autoCreateFields:true}"
-
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
- <processor class="solr.LogUpdateProcessorFactory"/>
- <processor class="solr.DistributedUpdateProcessorFactory"/>
- <processor class="solr.RunUpdateProcessorFactory"/>
- </updateRequestProcessorChain>
<!-- Deduplication
@@ -1273,7 +1239,6 @@
<!-- The following response writers are implicitly configured unless
overridden...
-->
- <!--
<queryResponseWriter name="xml"
default="true"
class="solr.XMLResponseWriter" />
@@ -1284,7 +1249,6 @@
<queryResponseWriter name="phps"
class="solr.PHPSerializedResponseWriter"/>
<queryResponseWriter name="csv"
class="solr.CSVResponseWriter"/>
<queryResponseWriter name="schema.xml"
class="solr.SchemaXmlResponseWriter"/>
- -->
<queryResponseWriter name="json"
class="solr.JSONResponseWriter">
<!-- For the purposes of the tutorial, JSON responses are written as
@@ -1293,7 +1257,7 @@
-->
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
-
+
<!--
Custom response writers can be declared as needed...
-->
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20181208/c88cdb8e/attachment-0001.html>
After some testsing, I managed to get proper functionning
- The schema.xml is attached below (quite different from the one
provided on teh wiki) (in bold the core differences) (NGramFilterFactory
is the class that replace the fts_squat "partial=3 full=15",
everything
else is just a big hammer to smash a tiny fly)
- One need to remove the "managed-schema" file in the
{prefix}/server/solr/dovecot/conf.
- One need to remove everything under {prefix}/server/solr/dovecot/data/
- The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see
diff)
- Restart Solr
- my dovecot.conf is :
fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_decoder = decode2text
fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/
--- schema.xml
<?xml version="1.0" encoding="UTF-8"?>
<schema name="dovecot" version="2.0">
<uniqueKey>id</uniqueKey>
<FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" />
<FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" />
<FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" />
<fieldType name="text" class="solr.TextField"
autoGeneratePhraseQueries="true"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory"
words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1"
splitOnNumerics="1" catenateWords="1"
catenateNumbers="1"
catenateAll="1"/>
<filter class="solr.FlattenGraphFilterFactory"/> <!--
required on index
analyzers after graph filters -->
<filter class="solr.LowerCaseFilterFactory"/>
<FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3"
MAXGRAMSIZE="15"
/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymGraphFilterFactory"
expand="true"
ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/> <!--
required on index
analyzers after graph filters -->
<filter class="solr.StopFilterFactory"
words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1"
splitOnCaseChange="1"
splitOnNumerics="1" catenateWords="1"
catenateNumbers="1"
catenateAll="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3"
MAXGRAMSIZE="15"
/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="_version_" type="long"
indexed="true" stored="true"/>
<field name="bcc" type="text" indexed="true"
stored="false"/>
<field name="body" type="text" indexed="true"
stored="false"/>
<field name="box" type="string" indexed="true"
required="true"
stored="true"/>
<field name="cc" type="text" indexed="true"
stored="false"/>
<field name="from" type="text" indexed="true"
stored="false"/>
<field name="hdr" type="text" indexed="true"
stored="false"/>
<field name="id" type="string" indexed="true"
required="true"
stored="true"/>
<field name="subject" type="text"
indexed="true" stored="false"/>
<field name="to" type="text" indexed="true"
stored="false"/>
<field name="uid" type="long" indexed="true"
required="true"
stored="true"/>
<field name="user" type="string" indexed="true"
required="true"
stored="true"/>
</schema>
------ diff solrconfig
--- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505
+0000
+++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000
@@ -1128,6 +1128,7 @@
See http://wiki.apache.org/solr/GuessingFieldTypes
-->
+ <schemaFactory
class="ClassicIndexSchemaFactory"></schemaFactory>
<updateProcessor class="solr.UUIDUpdateProcessorFactory"
name="uuid"/>
<updateProcessor
class="solr.RemoveBlankFieldUpdateProcessorFactory"
name="remove-blank"/>
<updateProcessor
class="solr.FieldNameMutatingUpdateProcessorFactory"
name="field-name-mutating">
@@ -1158,43 +1159,8 @@
<str>yyyy-MM-dd</str>
</arr>
</updateProcessor>
- <updateProcessor
class="solr.AddSchemaFieldsUpdateProcessorFactory"
name="add-schema-fields">
- <lst name="typeMapping">
- <str name="valueClass">java.lang.String</str>
- <str name="fieldType">text_general</str>
- <lst name="copyField">
- <str name="dest">*_str</str>
- <int name="maxChars">256</int>
- </lst>
- <!-- Use as default mapping instead of defaultFieldType -->
- <bool name="default">true</bool>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Boolean</str>
- <str name="fieldType">booleans</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.util.Date</str>
- <str name="fieldType">pdates</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Long</str>
- <str name="valueClass">java.lang.Integer</str>
- <str name="fieldType">plongs</str>
- </lst>
- <lst name="typeMapping">
- <str name="valueClass">java.lang.Number</str>
- <str name="fieldType">pdoubles</str>
- </lst>
- </updateProcessor>
<!-- The update.autoCreateFields property can be turned to false to
disable schemaless mode -->
- <updateRequestProcessorChain
name="add-unknown-fields-to-the-schema"
default="${update.autoCreateFields:true}"
-
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
- <processor class="solr.LogUpdateProcessorFactory"/>
- <processor class="solr.DistributedUpdateProcessorFactory"/>
- <processor class="solr.RunUpdateProcessorFactory"/>
- </updateRequestProcessorChain>
<!-- Deduplication
@@ -1273,7 +1239,6 @@
<!-- The following response writers are implicitly configured unless
overridden...
-->
- <!--
<queryResponseWriter name="xml"
default="true"
class="solr.XMLResponseWriter" />
@@ -1284,7 +1249,6 @@
<queryResponseWriter name="phps"
class="solr.PHPSerializedResponseWriter"/>
<queryResponseWriter name="csv"
class="solr.CSVResponseWriter"/>
<queryResponseWriter name="schema.xml"
class="solr.SchemaXmlResponseWriter"/>
- -->
<queryResponseWriter name="json"
class="solr.JSONResponseWriter">
<!-- For the purposes of the tutorial, JSON responses are written as
@@ -1293,7 +1257,7 @@
-->
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
-
+
<!--
Custom response writers can be declared as needed...
-->
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20181208/4b09e842/attachment.html>