dovecot - Feb 2021 - fts_encoder

On 08/02/2021 23:05, Stuart Henderson wrote:
> On 2021/02/08 21:33, Joan Moreau wrote:
>> Yes , once again : output of the decoder is fine, I also put log inide
the dovecot core to
>> check whether data is properly transmitted, and result is that it is
(i.e. dovecot core
>> receives the proper output of pdftotext via the decoder
>>
>> Now, that data is the /not/ the one sent from dovecot core to the fts
plugin (and this is the
>> same issue for solr and all other plugins)
> Seems that something is different with your setup than John's and mine
> then, as fts_solr rawlog (which is just the http request split into
> .in and .out files) has the decoded file for us.
>
> Did you try with the actual fts_solr plugin so it's a direct comparison
> with what we see? There is no need for a real solr server, just point it
> at any http server (or I guess netcat listening on a port will also do)
> with
>
> mail_plugins = fts fts_solr
>
> plugin {
>   fts_autoindex = yes
>   fts = solr
>   fts_solr = url=http://127.0.0.1:80/ rawlog_dir=/tmp/solr
> }
>
> If that is not showing decoded for you then I suppose there's some
> problem on the way into/through fts. And if it does show as decoded
> then perhaps fts_solr is doing something slightly different than the
> places you're examining in fts and your plugin, and that might give
> a point to work backwards from.
>
I'd also recommend Joan to look into some of the potential configuration
issues I mentioned in my first reply and if the problem persists, post
some clear evidence.

John

If I place the following code in the plugin 
fts_backend_xxx_update_build_more function (lucene, squat and xapian, as 
solr refuses to work properly on my setup)

         {
                 char * s = i_strdup("EMPTY");
                 if(data != NULL) { i_free(s); s = i_strndup(data,20); }
                 i_info("fts_backend_update_build_more: data like 
'%s'",s);
                 i_free(s);
         }

and if I send a PDF by email, the data shown in the log is "%PDF-1.7 "

so it does mean the decoder data is not properly transmitted to the 
plugin

Something is wrong in the data transmission

On 2021-02-09 11:58, John Fawcett wrote:
> On 08/02/2021 23:05, Stuart Henderson wrote: On 2021/02/08 21:33, Joan 
> Moreau wrote: Yes , once again : output of the decoder is fine, I also 
> put log inide the dovecot core to
> check whether data is properly transmitted, and result is that it is 
> (i.e. dovecot core
> receives the proper output of pdftotext via the decoder
> 
> Now, that data is the /not/ the one sent from dovecot core to the fts 
> plugin (and this is the
> same issue for solr and all other plugins) Seems that something is 
> different with your setup than John's and mine
> then, as fts_solr rawlog (which is just the http request split into
> .in and .out files) has the decoded file for us.
> 
> Did you try with the actual fts_solr plugin so it's a direct comparison
> with what we see? There is no need for a real solr server, just point 
> it
> at any http server (or I guess netcat listening on a port will also do)
> with
> 
> mail_plugins = fts fts_solr
> 
> plugin {
> fts_autoindex = yes
> fts = solr
> fts_solr = url=http://127.0.0.1:80/ rawlog_dir=/tmp/solr
> }
> 
> If that is not showing decoded for you then I suppose there's some
> problem on the way into/through fts. And if it does show as decoded
> then perhaps fts_solr is doing something slightly different than the
> places you're examining in fts and your plugin, and that might give
> a point to work backwards from.
  I'd also recommend Joan to look into some of the potential 
configuration
issues I mentioned in my first reply and if the problem persists, post
some clear evidence.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210209/9b500f91/attachment-0001.html>