search for: fts_decoder

...cot/decode2text socket is created with sufficiant permissions (rw for anybody), but decode2text.sh is not started (it's rx for anybody). When creating lucene indexes, I cannot see it executed either (and no attachment-only content is findable afterwards). No errors with mail_debug - if I change fts_decoder to a nonexistant value, that's logged. Lutz

Hello, I have configure dovecot with solr and I wanted to let solr index content of attachments. For testing I have used biabam command line tool to generate emails with attachments. I have found that dovecot with fts_decoder incorrectly decodes these attachments from biabam and therefore pdftotext has reported corrupted PDF. The problem is that biabam generates header with charset=binary and dovecot message decoder tries to process it as UTF8 or non-UTF8 data. ========================================================...

...or SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D ) I added the fts_enforce following AKi advice. I removed fts_decoder for the time being. I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors. Bottom line, I am back to squat, but as it is not maintained so crashed also times to times. I think we should discuss on (1) Wh...

Hello, is it possible to use the Solr full text search plugin for indexing mail attachments? I found a very old patch and some hints regarding a fts_decoder script that I don't understand. Making Solr indexing PDF or Office files shouldn't be that difficult, but how can I enable the plugin to transfer the attachments to Solr? Best regards, Sebastian

...t a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D ) > > I added the fts_enforce following AKi advice. > > I removed fts_decoder for the time being. > > I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors. > > Bottom line, I am back to squat, but as it is not maintained so crashed also times to times. > > I think...

here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x) <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <types> <fieldType name="string" class="solr.StrField" /> <fieldType name="gjlong"

...ot maintaining squat), and it > leads to totally funny results (for instance, I type "emirates" in my > "Air Companies" subfolder and get a lot of results .. but of competing > companies :D ) > > I added the fts_enforce following AKi advice. > > I removed fts_decoder for the time being. > > I don't know where to go now. Dovcot still returning errors and SOlr > still companinig with "Out of range" and other Java errors. > > Bottom line, I am back to squat, but as it is not maintained so > crashed also times to times. > > &...

...t a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D ) > > I added the fts_enforce following AKi advice. > > I removed fts_decoder for the time being. > > I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors. > > Bottom line, I am back to squat, but as it is not maintained so crashed also times to times. > > I think...

...these matters - but I do have a working setup and I'd like to help you if I can.? If you're willing to - take a deep breath and let's try starting over. Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced".? I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify. I suggest you try once more: delete Solr completely.? Re-install per the directions and use *my* managed-schema.? Also comment out the Dovecot dire...

...666 >>>> ? } >>>> } >>>> >>>> And my 90-plugins.cof: >>>> ? # Add dovecot fts config >>>> ? fts = solr >>>> ? fts_autoindex = no >>>> ? fts_solr = url=http://localhost:8080/solr/ debug >>>> ? fts_decoder = decode2text >>>> >>>> Am I doing anything wrong? >>>> >>>> Thanks for your help. >>>> >>>> Andr? >>> >>> hello again, >>> >>> The issue is not solved, but I have been able to see an error m...

...uot; file in the {prefix}/server/solr/dovecot/conf. - One need to remove everything under {prefix}/server/solr/dovecot/data/ - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) - Restart Solr - my dovecot.conf is : fts = solr fts_autoindex = yes fts_enforced = yes fts_decoder = decode2text fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ --- schema.xml <?xml version="1.0" encoding="UTF-8"?> <schema name="dovecot" version="2.0"> <uniqueKey>id</uniqueKey> <FIELDTYPE NAME="STRING" CLA...

...m antispam_crm_notspam_arg = --good antispam_crm_spam_arg = --spam antispam_signature = X-CRM114-CacheID antispam_spam_pattern_ignorecase = crm-spam* antispam_trash_pattern_ignorecase = Trash* antispam_unsure_pattern_ignorecase = crm-unsure* fts = solr fts_autoindex = yes fts_decoder = decode2text fts_solr = url=http://192.168.119.5:8983/solr/ break-imap-search mail_log_events = delete undelete expunge copy mail_log_fields = uid box msgid size mail_log_group_events = quota = dict:::redis:host=127.0.0.1:prefix=user/ quota_rule2 = Trash:storage=+20%% quota_wa...

...difficult to implement well. 8. Index attachments separately, so it would be possible to search only attachments. (Should "SEARCH BODY word1 BODY word2" return matches if word1 and word2 are in different attachments?) 9. Attachments can be translated to indexable UTF-8 text already with fts_decoder setting by doing it via a conversion script. This could also support Apache Tika server directly. 10. It should be configurable which fields are indexed. Body and header would always be separately indexed. Optionally there could be also at least: attachments, From, To, Cc, Bcc and Subject. The Fro...

Hi, I've seen that latest Dovecot has the ability to use the Apache Tika Server to extract metadata from attachments. That is a pretty useful feature and looks to be an improvement over piping the attachment through a shellscript as done with fts_decoder. Having this feature enabled though I sometimes do see the situation that the tika service does not reply in time and it looks like the whole FTS transaction gets aborted: dovecot: indexer-worker(eggs at localhost): Error: fts_tika: PUT http://localhost:9998/tika/ failed: Request timed out (Re...

...;>>> } >>>>> >>>>> And my 90-plugins.cof: >>>>> ? # Add dovecot fts config >>>>> ? fts = solr >>>>> ? fts_autoindex = no >>>>> ? fts_solr = url=http://localhost:8080/solr/ debug >>>>> ? fts_decoder = decode2text >>>>> >>>>> Am I doing anything wrong? >>>>> >>>>> Thanks for your help. >>>>> >>>>> Andr? >>>> >>>> hello again, >>>> >>>> The issue is not solved...

...= dovecot >> unix_listener decode2text { >> mode = 0666 >> } >> } >> >> And my 90-plugins.cof: >> # Add dovecot fts config >> fts = solr >> fts_autoindex = no >> fts_solr = url=http://localhost:8080/solr/ debug >> fts_decoder = decode2text >> >> Am I doing anything wrong? >> >> Thanks for your help. >> >> Andr? > > hello again, > > The issue is not solved, but I have been able to see an error message: > indexer-worker(andre): Error: parser script didn't send an...

...edb;-f > antispam_mail_spam = -s > antispam_spam = SPAM;Spam;Junk > antispam_trash = Trash > antispam_unsure = Filtered-Spam;Possible-Spam > fetchmail_pidfile = %h/.fetchmail.pid > fts = lucene > fts_autoindex = yes > fts_autoindex_max_recent_msgs = 30 > fts_decoder = decode2text > fts_index_timeout = 60 > fts_lucene = whitespace_chars=@. mime_parts > mailbox_alias_new = Sent Messages > mailbox_alias_new2 = Sent Items > mailbox_alias_old = Sent > mailbox_alias_old2 = Sent > quota = maildir:User quota > quota_grace = 10%%...

...r.conf: service decode2text { executable = script /usr/local/bin/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } } And my 90-plugins.cof: # Add dovecot fts config fts = solr fts_autoindex = no fts_solr = url=http://localhost:8080/solr/ debug fts_decoder = decode2text Am I doing anything wrong? Thanks for your help. Andr?

...One need to remove everything under {prefix}/server/solr/dovecot/data/ > > - The {prefix}/server/solr/dovecot/conf/solrconfig.xml is as below (see diff) > > - Restart Solr > > - my dovecot.conf is : > > fts = solr > fts_autoindex = yes > fts_enforced = yes > fts_decoder = decode2text > > fts_solr = url=http://(SOLR SERVER):8983/solr/dovecot/ > > --- schema.xml > > <?xml version="1.0" encoding="UTF-8"?> > <schema name="dovecot" version="2.0"> > <uniqueKey>id</uniqueKey> &g...

THen the Squat shall be maintained until the SOlr plugin is upgraded, as Squat does resolve the problem (fts, partial search, etc...) On 2018-12-05 12:20, Aki Tuomi wrote: > It seems we forgot to document that "break-imap-search" was dropped in v2.3. That has now been updated. > > Also Solr does not support prefix/substring search unless you configure solr to support it.