dovecot - Nov 2020 - v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

On 11/1/20 10:35 AM, John Fawcett wrote:
> Yes, getting more data about any potential problem would be useful.
> 
> Just to clarify: I have a fully working search setup for some time now
> over various dovecot releases, so no patches needed to get it working.
> 
> My setup does use fts plugin and fts-solr plugin, but it does not use
> lib-fts functionality (that has many features for example it was
> stopping you indexing excluded words like tambien). On my setup without
> lib-fts everything goes to solr which does the work of indexing without
> all the features of lib-fts.
withOUT libfts

-	fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/ use_libfts
soft_commit=yes batch_size=250
+	fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/
soft_commit=yes batch_size=250

and unmodified dovecot-provided schema/config,

	/bin/cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml
/path/to/solr/data/dovecot/conf/solrconfig.xml
	/bin/cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml
/path/to/solr/data/dovecot/conf/schema.xml

i suspect my config's now more similar to yours.

checking,

	doveadm fts rescan -u testuser at example.com
	doveadm index      -u testuser at example.com -q '*'

as before

	doveadm fts lookup -u testuser at example.com body "t?sting"

panics,

	doveadm(testuser at example.com): Panic: file mail-storage.c: line 2112
(mailbox_get_open_status): assertion failed: (box->opened)
	doveadm(testuser at example.com): Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7f7829b81cc6] ->
/usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f7829b81de2] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x10025b) [0x7f7829b8b25b] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x100297) [0x7f7829b8b297] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x59bc6) [0x7f7829ae4bc6] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f7829c9879e] ->
/usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849) [0x7f78296ea849] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51) [0x7f782930b7c1]
-> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280)
[0x7f78270d0280] -> doveadm(+0x343cd) [0x55aa57edc3cd] ->
doveadm(+0x34fe0) [0x55aa57edcfe0] ->
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55aa57edde2d] ->
doveadm(doveadm_cmd_run_ver2+0x4e8) [0x55aa57eee8d8] ->
doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55aa57eee92e] ->
doveadm(main+0x1d4) [0x55aa57ecccf4] ->
/lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7829746042] ->
doveadm(_start+0x2e) [0x55aa57ecd1ce]
	Aborted

but search, even for accented characters,

	doveadm search -u testuser at example.com subject "t?sting"
		42d73837f133a05fad4d0000f8839f03 1
		813ef60e984f1b5f5fc200005439fba4 293

	doveadm search -u testuser at example.com body "t?sting"
		ba899d0cfe33a05fbe4d0000f8839f03 1
		813ef60e984f1b5f5fc200005439fba4 293

appears to work.

next, to get tokenization -- at least email/url (UAX29URLEmailTokenizer) -- and
lowercase & icu normalization working and verified.

sending to the list as well

On 02/11/2020 17:40, PGNet Dev wrote:
> On 11/1/20 10:35 AM, John Fawcett wrote:
>> Yes, getting more data about any potential problem would be useful.
>>
>> Just to clarify: I have a fully working search setup for some time now
>> over various dovecot releases, so no patches needed to get it working.
>>
>> My setup does use fts plugin and fts-solr plugin, but it does not use
>> lib-fts functionality (that has many features for example it was
>> stopping you indexing excluded words like tambien). On my setup without
>> lib-fts everything goes to solr which does the work of indexing without
>> all the features of lib-fts.
>
> withOUT libfts
>
> -??? fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
> +??? fts_solr = url=https://solr.presence-group.net:8984/solr/dovecot/
> soft_commit=yes batch_size=250
>
> and unmodified dovecot-provided schema/config,
>
> ????/bin/cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml
> /path/to/solr/data/dovecot/conf/solrconfig.xml
> ????/bin/cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml
> /path/to/solr/data/dovecot/conf/schema.xml
>
> i suspect my config's now more similar to yours.
>
> checking,
>
> ????doveadm fts rescan -u testuser at example.com
> ????doveadm index????? -u testuser at example.com -q '*'
>
> as before
>
> ????doveadm fts lookup -u testuser at example.com body "t?sting"
>
> panics,
>
> ????doveadm(testuser at example.com): Panic: file mail-storage.c: line
> 2112 (mailbox_get_open_status): assertion failed: (box->opened)
> ????doveadm(testuser at example.com): Error: Raw backtrace:
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46)
> [0x7f7829b81cc6] ->
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22)
> [0x7f7829b81de2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10025b)
> [0x7f7829b8b25b] -> /usr/lib64/dovecot/libdovecot.so.0(+0x100297)
> [0x7f7829b8b297] -> /usr/lib64/dovecot/libdovecot.so.0(+0x59bc6)
> [0x7f7829ae4bc6] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f7829c9879e]
> -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849)
> [0x7f78296ea849] ->
> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51)
> [0x7f782930b7c1] ->
> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280)
> [0x7f78270d0280] -> doveadm(+0x343cd) [0x55aa57edc3cd] ->
> doveadm(+0x34fe0) [0x55aa57edcfe0] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55aa57edde2d]
> -> doveadm(doveadm_cmd_run_ver2+0x4e8) [0x55aa57eee8d8] ->
> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x55aa57eee92e] ->
> doveadm(main+0x1d4) [0x55aa57ecccf4] ->
> /lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7829746042] ->
> doveadm(_start+0x2e) [0x55aa57ecd1ce]
> ????Aborted
>
> but search, even for accented characters,
>
> ????doveadm search -u testuser at example.com subject "t?sting"
> ??????? 42d73837f133a05fad4d0000f8839f03 1
> ??????? 813ef60e984f1b5f5fc200005439fba4 293
>
> ????doveadm search -u testuser at example.com body "t?sting"
> ??????? ba899d0cfe33a05fbe4d0000f8839f03 1
> ??????? 813ef60e984f1b5f5fc200005439fba4 293
>
> appears to work.
>
> next, to get tokenization -- at least email/url
> (UAX29URLEmailTokenizer) -- and lowercase & icu normalization working
> and verified.
>
the panic on doveadm fts lookup is to be expected and solved by my
previous patch.

I think you've now got a config very similar to mine. One last check I
did was to search for the same string with the grave accent. e.g.
t?sting, so that should NOT be found. That was just proof that it was
actually searching for accents and not folding them to e.

I saw Aki's advice about removing use_tls because of issues, and I know
I'm repeating myself here :-) but so far I have seen no concrete
evidence of issues on your original setup. Doesn't mean there aren't,
but until there is evidence, can't do much towards solving them.

John

John

On 11/2/20 9:03 AM, John Fawcett wrote:
> the panic on doveadm fts lookup is to be expected and solved by my
> previous patch.
snip

all noted.

atm, I've managed to get solr-backend indexing, and cmd-line searchable with
doveadm, correctly finding the unfolded, accented characters in my
'test'.
so for my 'admin' needs, that's good enuf for now.

waiting for your patch to get picked up into what seems like a
new-release-coming-soon(ish) seems a good idea.  i'll likely spin up a
dovecot/master DIY-build to explore, anyway.

and, it's all somewhat moot for end-users using TBird in that its backend
server-side search is broken anyway.

time to take a pause and leave it alone for a bit.



thx!