THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...) On 2018-12-05 12:20, Aki Tuomi wrote:> It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated. > > Also Solr does not support prefix/substring search unless you configure solr to support it. > > Aki > > On 5.12.2018 11.50, Joan Moreau wrote: > > Well, "break-imap-search" option prevent dovecot from starting. > > I used solr on one server since yesterday. > > First feedback is that > > * BODY search is not working > > * FTS is not really working in headers : For instance, if you search "kliinik" but you type "kliini" (missing one k), the search does not return anything , whereas it shall return more results (as the keyword is smaller) that the intended search > > On 2018-12-05 07:55, Aki Tuomi wrote: > > Full text search gets enabled automatically, although your probably want to set fts_enforced=yes to make sure it's used for all kinds of things automatically. break-imap-search is supported, but does nothing. > > Updated the wiki a bit. > > Aki > > On 4.12.2018 20.50, Joan Moreau via dovecot wrote: > > also, the value "break-imap-search" is not supported in fts_solr parameter, contrary to the wiki > > and not sure then how to enable full text search in BODY > > On 2018-12-04 19:40, Joan Moreau via dovecot wrote: > > In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : > > sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance > > before updating the schema.xml . > > Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users > > Additionaly, the url is http://(solr_ [1]server):8983/solr/dovecot/ (error in wiki) > > On 2018-12-04 19:01, Joan Moreau via dovecot wrote: > > Hi > > I am giving Solr another try. > > Using Solr 7.5 (from Archlinux AUR), I do not see anymore the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per https://wiki.dovecot.org/Plugins/FTS/Solr) > > Maybe some things have moved ? > > Thank you so muchLinks: ------ [1] http://(solr -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/64dc8d61/attachment.html>
Squat has not been maintained since 2.2.something, and shall not be maintained. Sorry. If you want, you can become maintainer for squat plugin. Aki On 5.12.2018 13.57, Joan Moreau wrote:> > THen the Squat shall be maintained until the SOlr plugin is upgraded, > as Squat does resolve the problem (fts, partial search, etc...) > > > ? > > > On 2018-12-05 12:20, Aki Tuomi wrote: > >> It seems we forgot to document that "break-imap-search" was dropped >> in v2.3. That has now been updated. >> >> Also Solr does not support prefix/substring search unless you >> configure solr to support it. >> >> Aki >> >> On 5.12.2018 11.50, Joan Moreau wrote: >>> >>> Well, "break-imap-search" option prevent dovecot from starting. >>> >>> >>> I used solr on one server since yesterday. >>> >>> >>> First feedback is that >>> >>> * BODY search is not working >>> >>> * FTS is not really working in headers : For instance, if you search >>> "kliinik" but you type "kliini" (missing one k), the search does not >>> return anything , whereas it shall return more results (as the >>> keyword is smaller) that the intended search >>> >>> >>> ? >>> >>> >>> On 2018-12-05 07:55, Aki Tuomi wrote: >>> >>> Full text search gets enabled automatically, although your >>> probably want to set fts_enforced=yes to make sure it's used for >>> all kinds of things automatically. break-imap-search is >>> supported,? but does nothing. >>> >>> Updated the wiki a bit. >>> >>> Aki >>> >>> On 4.12.2018 20.50, Joan Moreau via dovecot wrote: >>> >>> also, the value "break-imap-search" is not supported in >>> fts_solr parameter, contrary to the wiki >>> >>> and not sure then how to enable full text search in BODY >>> >>> ? >>> >>> >>> On 2018-12-04 19:40, Joan Moreau via dovecot wrote: >>> >>> In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr >>> ), it would nice to stipulate to the reader? to type the >>> command : >>> >>> sudo -u solr /opt/solr/bin/solr create -c dovecot # to >>> create the dovecot instance >>> >>> before updating the schema.xml . >>> >>> Also,? schema.xml is in >>> /opt/solr/server/solr/dovecot/conf for archlinux users >>> >>> Additionaly, the url is http://(solr_ >>> <http://(solr>server):8983/solr/dovecot/ (error in wiki) >>> >>> >>> ? >>> >>> >>> On 2018-12-04 19:01, Joan Moreau via dovecot wrote: >>> >>> Hi >>> >>> I am giving Solr another try. >>> >>> Using Solr 7.5 (from Archlinux AUR), I do not see >>> anymore the"solr/conf/schema.xml" neither the >>> "managed-schema" folder. (as >>> per?https://wiki.dovecot.org/Plugins/FTS/Solr)? >>> >>> Maybe some things have moved ? >>> >>> Thank you so much >>> >>> >>>-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/424244e6/attachment-0001.html>
Additionally, I get "Invalid search response" on my Android , using dovecot-solr, whereas squat works fine Why making squat obsolete ? It is simple and straightforward On December 5, 2018 12:57:22 PM Joan Moreau via dovecot <dovecot at dovecot.org> wrote:> THen the Squat shall be maintained until the SOlr plugin is upgraded, as > Squat does resolve the problem (fts, partial search, etc...) > > > > On 2018-12-05 12:20, Aki Tuomi wrote: >> It seems we forgot to document that "break-imap-search" was dropped in >> v2.3. That has now been updated. >> Also Solr does not support prefix/substring search unless you configure >> solr to support it. >> Aki >> On 5.12.2018 11.50, Joan Moreau wrote: >>> Well, "break-imap-search" option prevent dovecot from starting. >>> >>> I used solr on one server since yesterday. >>> >>> First feedback is that >>> * BODY search is not working >>> * FTS is not really working in headers : For instance, if you search >>> "kliinik" but you type "kliini" (missing one k), the search does not return >>> anything , whereas it shall return more results (as the keyword is smaller) >>> that the intended search >>> >>> >>> >>> On 2018-12-05 07:55, Aki Tuomi wrote: >>> Full text search gets enabled automatically, although your probably want to >>> set fts_enforced=yes to make sure it's used for all kinds of things >>> automatically. break-imap-search is supported, but does nothing. >>> Updated the wiki a bit. >>> Aki >>> On 4.12.2018 20.50, Joan Moreau via dovecot wrote: >>> also, the value "break-imap-search" is not supported in fts_solr parameter, >>> contrary to the wiki >>> and not sure then how to enable full text search in BODY >>> >>> >>> On 2018-12-04 19:40, Joan Moreau via dovecot wrote: >>> In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice >>> to stipulate to the reader to type the command : >>> sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot >>> instance >>> before updating the schema.xml . >>> Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users >>> Additionaly, the url is http://(solr_server):8983/solr/dovecot/ (error in wiki) >>> >>> >>> >>> On 2018-12-04 19:01, Joan Moreau via dovecot wrote: >>> Hi >>> I am giving Solr another try. >>> Using Solr 7.5 (from Archlinux AUR), I do not see anymore >>> the"solr/conf/schema.xml" neither the "managed-schema" folder. (as per >>> https://wiki.dovecot.org/Plugins/FTS/Solr) >>> Maybe some things have moved ? >>> Thank you so much-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181205/628d74ec/attachment.html>
On Wed Dec 05 2018 07:35:39 GMT-0500 (Eastern Standard Time), Joan Moreau via dovecot <dovecot at dovecot.org> wrote:> Why making squat obsolete ? It is simple and straightforwardBecause no one has stepped up to maintain it?
After some testsing, I managed to get proper functionning - The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly) - One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf. - One need to remove everything under {prefix}/server/solr/dovecot/data/ - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - my dovecot.conf is : fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ --- schema.xml <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" /> <FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" /> <FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema> ------ diff solrconfig --- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@ See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor> <!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain> <!-- Deduplication @@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - --> <queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... --> -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181208/c88cdb8e/attachment-0001.html>
After some testsing, I managed to get proper functionning - The schema.xml is attached below (quite different from the one provided on teh wiki) (in bold the core differences) (NGramFilterFactory is the class that replace the fts_squat "partial=3 full=15", everything else is just a big hammer to smash a tiny fly) - One need to remove the "managed-schema" file in the {prefix}/server/solr/dovecot/conf. - One need to remove everything under {prefix}/server/solr/dovecot/data/ - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - Restart Solr - my dovecot.conf is : fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ --- schema.xml <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <FIELDTYPE NAME="STRING" CLASS="SOLR.STRFIELD" /> <FIELDTYPE NAME="LONG" CLASS="SOLR.TRIELONGFIELD" /> <FIELDTYPE NAME="BOOLEAN" CLASS="SOLR.BOOLFIELD" /> <fieldType name="text" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> <filter class="solr.LowerCaseFilterFactory"/> <FILTER CLASS="SOLR.NGRAMFILTERFACTORY" MINGRAMSIZE="3" MAXGRAMSIZE="15" /> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="bcc" type="text" indexed="true" stored="false"/> <field name="body" type="text" indexed="true" stored="false"/> <field name="box" type="string" indexed="true" required="true" stored="true"/> <field name="cc" type="text" indexed="true" stored="false"/> <field name="from" type="text" indexed="true" stored="false"/> <field name="hdr" type="text" indexed="true" stored="false"/> <field name="id" type="string" indexed="true" required="true" stored="true"/> <field name="subject" type="text" indexed="true" stored="false"/> <field name="to" type="text" indexed="true" stored="false"/> <field name="uid" type="long" indexed="true" required="true" stored="true"/> <field name="user" type="string" indexed="true" required="true" stored="true"/> </schema> ------ diff solrconfig --- /data/backup/solr/solrconfig.xml.joan 2018-12-08 14:31:47.716344505 +0000 +++ solrconfig.xml 2018-12-08 15:36:28.948267225 +0000 @@ -1128,6 +1128,7 @@ See http://wiki.apache.org/solr/GuessingFieldTypes --> + <schemaFactory class="ClassicIndexSchemaFactory"></schemaFactory> <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/> <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/> <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> @@ -1158,43 +1159,8 @@ <str>yyyy-MM-dd</str> </arr> </updateProcessor> - <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> - <lst name="typeMapping"> - <str name="valueClass">java.lang.String</str> - <str name="fieldType">text_general</str> - <lst name="copyField"> - <str name="dest">*_str</str> - <int name="maxChars">256</int> - </lst> - <!-- Use as default mapping instead of defaultFieldType --> - <bool name="default">true</bool> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Boolean</str> - <str name="fieldType">booleans</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.util.Date</str> - <str name="fieldType">pdates</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Long</str> - <str name="valueClass">java.lang.Integer</str> - <str name="fieldType">plongs</str> - </lst> - <lst name="typeMapping"> - <str name="valueClass">java.lang.Number</str> - <str name="fieldType">pdoubles</str> - </lst> - </updateProcessor> <!-- The update.autoCreateFields property can be turned to false to disable schemaless mode --> - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" - processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> - <processor class="solr.LogUpdateProcessorFactory"/> - <processor class="solr.DistributedUpdateProcessorFactory"/> - <processor class="solr.RunUpdateProcessorFactory"/> - </updateRequestProcessorChain> <!-- Deduplication @@ -1273,7 +1239,6 @@ <!-- The following response writers are implicitly configured unless overridden... --> - <!-- <queryResponseWriter name="xml" default="true" class="solr.XMLResponseWriter" /> @@ -1284,7 +1249,6 @@ <queryResponseWriter name="phps" class="solr.PHPSerializedResponseWriter"/> <queryResponseWriter name="csv" class="solr.CSVResponseWriter"/> <queryResponseWriter name="schema.xml" class="solr.SchemaXmlResponseWriter"/> - --> <queryResponseWriter name="json" class="solr.JSONResponseWriter"> <!-- For the purposes of the tutorial, JSON responses are written as @@ -1293,7 +1257,7 @@ --> <str name="content-type">text/plain; charset=UTF-8</str> </queryResponseWriter> - + <!-- Custom response writers can be declared as needed... --> -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20181208/4b09e842/attachment.html>
Possibly Parallel Threads
- SolrCore 'dovecot' is not available due to init failure: fieldType 'text_general' not found in the schema
- Solr
- Solr
- SolrCore 'dovecot' is not available due to init failure: fieldType 'text_general' not found in the schema
- SolrCore 'dovecot' is not available due to init failure: fieldType 'text_general' not found in the schema