dovecot - Feb 2021 - fts_encoder

Well, in the function xxx_build_more of FTS plugin, the data received in 
the original PDF, not the output of pdftotext

Can you clarify where do you put your log in the solr plugin , so I can 
check the situation in the xapian plugin ?

On 2021-02-08 17:34, John Fawcett wrote:
> On 08/02/2021 15:22, Joan Moreau wrote:
> 
>> Well, thank you for the answer, but the actual issue is that data sent
>> by the decoder (stipulated in the conf file) is properly collected by
>> dovecot core, but /not/ sent to the plugin : the plugin receives the
>> original data.
>> 
>> This is not linked to a particular plugin (xapian, solr, squat, etc..)
>> but seems to be a general issue of dovecot core
> 
> Hi Joan
> 
> as far as I can see there's not a general issue in the dovecot core 
> with
> using the decoder. It works for me. I see the text extracted from PDF
> sent to solr (I enable raw_log feature to see the actual data going 
> over
> ) Also when I query solr I get a search hit for attachment text.
> 
> John
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210208/a3867b7b/attachment.html>

On 2021-02-08, Joan Moreau <jom at grosjo.net>
wrote:
> Well, in the function xxx_build_more of FTS plugin, the data received in 
> the original PDF, not the output of pdftotext
>
> Can you clarify where do you put your log in the solr plugin , so I can 
> check the situation in the xapian plugin ?
The log is particular to fts_solr, you set it with e.g.

"fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
rawlog_dir=/tmp/solr"

Confirmed it works for me, i.e. passes text from inside the pdf, and not
the whole pdf itself.

Did you check that decode2text.sh works ok on your system (when running
as the relevant uid)?

cat foo.pdf | sudo -u dovecot /usr/libexec/dovecot/decode2text.sh
application/pdf

On 08/02/2021 21:35, Joan Moreau wrote:
>
> Well, in the function xxx_build_more of FTS plugin, the data received
> in the original PDF, not the output of pdftotext
>
> Can you clarify where do you put your log in the solr plugin , so I
> can check the situation in the xapian plugin ?
>
I used the following setting in fts_solr parameter

rawlog_dir=<directory>

and made sure the directory was writeable by dovecot (777 just to be sure)

John