dovecot - Nov 2020 - v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

> On 01/11/2020 20:35 John Fawcett <john at voipsupport.it> wrote:
> 
>  
> On 01/11/2020 15:20, PGNet Dev wrote:
> > On 11/1/20 1:56 AM, John Fawcett wrote:
> >> At the moment I don't see other corrections needed in dovecot
apart from
> >> command line doveadm fts which is not a show stopper. Via doveadm
search
> >> I confirm - on my simple config - that search for accented or non
> >> accented characters works correctly as it does via imap
connection.
> >
> > thx.? hopefully it'll get considered for a next release soon.
> >
> >> Only thing I cannot vouch for is bringing dovecot fts library and
config
> >> into the equation because my setup delegates almost everything to
solr.
> >
> > do i understand correctly that you're solr-indexing your dovecot
mail
> > store withOUT using dovecot fts plugin, and that -- with your
> > aforementioned patch -- doveadm successfully uses the resulting
indexes?
> >
> > i hadn't yet seriously considered _circumventing_ fts plugin; if
this^
> > does get resolved soonish, then it's not a big deal.? if not, an
> > fts-plugin-less setup would be interesting to know more abt!
> >
> >> Can you get evidence of things not working? For example tests run
with
> >> soft_commit configured - that's important since without it the
updates
> >> don't show up immediately in searches, that do show that the
update is
> >> happening in solr via solr log, but then search is not working on
> >> accented characters, despite it working on other text in the same
> >> message? The solr logs also show whether the text was found or not
via
> >> the "hits=" value in the logged searches, for example:
> >>
> >> 2020-11-01 08:32:42.231 INFO? (qtp24119573-21) [?? x:dovecot]
> >> o.a.s.c.S.Request [dovecot]? webapp=/solr path=/select
> >>
params={q={!lucene+q.op%3DAND}body:tambi?n&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e54100000c54a863a+%2Buser:john
at voipsupport.it&rows=3202&wt=xml}
> >>
> >> hits=3 status=0 QTime=3
> >>
> >> But if no hits are found, then dovecot cannot be expected to
display
> >> results. It still may be an indexing problem though.
> >
> > my current config has soft_commit enabled,
> >
> > ? fts_solr = url=https://solr.example.com:8984/solr/dovecot/
> > use_libfts soft_commit=yes batch_size=250
> >
> > i'll see abt getting some clearer test results ...
> 
> Yes, getting more data about any potential problem would be useful.
> 
> Just to clarify: I have a fully working search setup for some time now
> over various dovecot releases, so no patches needed to get it working.
> 
> My setup does use fts plugin and fts-solr plugin, but it does not use
> lib-fts functionality (that has many features for example it was
> stopping you indexing excluded words like tambien). On my setup without
> lib-fts everything goes to solr which does the work of indexing without
> all the features of lib-fts.
> 
> My setup is like this not because of issues in lib-fts, but because I
> never had the need for it. There is no evidence at the moment however
> that even with lib-fts enabled there are issues with dovecot indexing or
> searching.
> 
> What is currently nor working is "doveadm fts" command line
utility. But
> this is mitigated by being able to use a similar command line utility
> "doveadm search". The issue on "dovecot fts" command
line utility has
> (so far as the available evidence suggests) no effect on indexing or
> imap searches.
> 
> fyi my working configuration includes fts and fts_solr plugins
> 
> mail_plugins =? quota notify replication fts fts_solr
> 
> (and those are also recalled in the various specific plugin settings for
> imap, lmtp ecc), The specific config I am using for fts and fts_solr is:
> 
> ? fts = solr
> ? fts_enforced = yes
> ? fts_solr = url=https://user at server.example.com:443/solr/dovecot/
> batch_size=500 soft_commit=no
> 
> BTW I use soft_commit=no because I have periodic soft commits setup on
> solr and I accept that newly indexed text won't become searchable for
up
> to that interval, but for your testing purposes much more useful as you
> have it.
> 
> John
PGNet,

you should try removing use_libfts from your config line and let solr do that
part.

Aki

On 11/2/20 12:44 AM, Aki Tuomi wrote:
> you should try removing use_libfts from your config line and let solr do
that part.
sry, i'm a bit confused.

you'd suggested I _add_ it,

	https://dovecot.org/pipermail/dovecot/2020-October/120258.html
> I can reproduce your problem with the `fts lookup` command. Luckily
it's equivalent to running `doveadm search`. I'll open a bug about this.
> Dovecot FTS tokenization is not done, unless you have `use_libfts` in
fts_solr setting, in your case
> fts_solr = url=https://solr.example.com:8984/solr/dovecot/ use_libfts
> Without this, everything is sent to to solr as-is, which is then expected
to do all the work.
So what's the recommendation? use use_libfts, or not?