dovecot - Oct 2020 - v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]

On 10/18/20 10:28 PM, Aki Tuomi wrote:
>> 	doveadm(myuser at example.com): Panic: file mail-storage.c: line 2112
(mailbox_get_open_status): assertion failed: (box->opened)
...
> I can reproduce your problem with the `fts lookup` command. Luckily
it's equivalent to running `doveadm search`. I'll open a bug about this.
Can you provide any status on the bug/fix?

Thanks.

On 31/10/2020 04:57, PGNet Dev wrote:
> On 10/18/20 10:28 PM, Aki Tuomi wrote:
>>> ????doveadm(myuser at example.com): Panic: file mail-storage.c:
line
>>> 2112 (mailbox_get_open_status): assertion failed: (box->opened)
> ...
>
>> I can reproduce your problem with the `fts lookup` command. Luckily
>> it's equivalent to running `doveadm search`. I'll open a bug
about this.
>
> Can you provide any status on the bug/fix?
>
> Thanks.
I can contribute a patch that solves the segfault. Unfortunately though
fts search may be more broken than this. It does not give me search
results, even though I see it querying solr and getting hits.

diff -ur dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c
dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c
--- dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c??? 2020-08-12
14:20:41.000000000 +0200
+++ dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c??? 2020-10-31
17:52:09.019388695 +0100
@@ -47,6 +47,14 @@
???? i_array_init(&result.scores, 16);
?
???? box = mailbox_alloc(info->ns->list, info->vname, 0);
+??? mailbox_set_reason(box,"fts search");
+??? if (mailbox_open(box) < 0) {
+??? ??? i_error("Couldn't open mailbox: %s",
+??? ??? ??? mailbox_get_last_internal_error(box, NULL));
+??? ??? doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
+??? ??? return -1;
+??? }
+
???? if (fts_backend_lookup(backend, box, ctx->search_args->args,
???? ??? ??? ??? ????? FTS_LOOKUP_FLAG_AND_ARGS, &result) < 0) {
???? ??? i_error("fts lookup failed");

On a more minor issue, with this patch if you search for a non existent
mailbox, it does give a segfault for a different assert, in mail-user.c
(*user)->refcount == 1.

doveadm(john at voipsupport.it): Error: Couldn't open mailbox: Mailbox
doesn't exist: inboxx
doveadm(john at voipsupport.it): Panic: file mail-user.c: line 229
(mail_user_deinit): assertion failed: ((*user)->refcount == 1)
doveadm(john at voipsupport.it): Error: Raw backtrace:
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f35c44c3ee2] ->
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7f35c44c3fee] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec53e)
[0x7f35c44ce53e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec581)
[0x7f35c44ce581] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7f35c44254ea] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0x56d87)
[0x7f35c47e4d87] -> doveadm(+0x2cb28) [0x55c0eaa57b28] ->
doveadm(+0x2d77c) [0x55c0eaa5877c] ->
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c0eaa5960d] ->
doveadm(doveadm_cmd_run_ver2+0x472) [0x55c0eaa6a372] ->
doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c0eaa6a497] ->
doveadm(main+0x1d4) [0x55c0eaa47c54] ->
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f35c3d34555] ->
doveadm(+0x1d0ef) [0x55c0eaa480ef]
Aborted

John

On 10/31/20 9:55 AM, John Fawcett wrote:
> I can contribute a patch that solves the segfault. Unfortunately though
> fts search may be more broken than this. It does not give me search
> results, even though I see it querying solr and getting hits.
Thx -- hopefully it moves this in the right direction.

Also on the 'good news' page, it appears there's been some progress
on Thunderbird's use of backend/server search,

	TBird "search on server" doesn't -- NO comm with backend
IMAP/SOLR; appears to be local-only search
	 https://bugzilla.mozilla.org/show_bug.cgi?id=1673928

	"A fix for this is upcoming."

Remains to be seen if the doveadm search issues, and implications on backend
problems, have any effect on the Thunderbird searches.

On 31/10/2020 17:55, John Fawcett wrote:
> On 31/10/2020 04:57, PGNet Dev wrote:
>> On 10/18/20 10:28 PM, Aki Tuomi wrote:
>>>> ????doveadm(myuser at example.com): Panic: file mail-storage.c:
line
>>>> 2112 (mailbox_get_open_status): assertion failed:
(box->opened)
>> ...
>>
>>> I can reproduce your problem with the `fts lookup` command. Luckily
>>> it's equivalent to running `doveadm search`. I'll open a
bug about this.
>> Can you provide any status on the bug/fix?
>>
>> Thanks.
> I can contribute a patch that solves the segfault. Unfortunately though
> fts search may be more broken than this. It does not give me search
> results, even though I see it querying solr and getting hits.
>
> diff -ur dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c
> dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c
> --- dovecot-2.3.11.3-orig/src/plugins/fts/doveadm-fts.c??? 2020-08-12
> 14:20:41.000000000 +0200
> +++ dovecot-2.3.11.3-patch/src/plugins/fts/doveadm-fts.c??? 2020-10-31
> 17:52:09.019388695 +0100
> @@ -47,6 +47,14 @@
> ???? i_array_init(&result.scores, 16);
> ?
> ???? box = mailbox_alloc(info->ns->list, info->vname, 0);
> +??? mailbox_set_reason(box,"fts search");
> +??? if (mailbox_open(box) < 0) {
> +??? ??? i_error("Couldn't open mailbox: %s",
> +??? ??? ??? mailbox_get_last_internal_error(box, NULL));
> +??? ??? doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
> +??? ??? return -1;
> +??? }
> +
> ???? if (fts_backend_lookup(backend, box, ctx->search_args->args,
> ???? ??? ??? ??? ????? FTS_LOOKUP_FLAG_AND_ARGS, &result) < 0) {
> ???? ??? i_error("fts lookup failed");
>
> On a more minor issue, with this patch if you search for a non existent
> mailbox, it does give a segfault for a different assert, in mail-user.c
> (*user)->refcount == 1.
>
> doveadm(john at voipsupport.it): Error: Couldn't open mailbox: Mailbox
> doesn't exist: inboxx
> doveadm(john at voipsupport.it): Panic: file mail-user.c: line 229
> (mail_user_deinit): assertion failed: ((*user)->refcount == 1)
> doveadm(john at voipsupport.it): Error: Raw backtrace:
> /usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
> [0x7f35c44c3ee2] ->
> /usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
> [0x7f35c44c3fee] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec53e)
> [0x7f35c44ce53e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xec581)
> [0x7f35c44ce581] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
> [0x7f35c44254ea] ->
> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x56d87)
> [0x7f35c47e4d87] -> doveadm(+0x2cb28) [0x55c0eaa57b28] ->
> doveadm(+0x2d77c) [0x55c0eaa5877c] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c0eaa5960d] ->
> doveadm(doveadm_cmd_run_ver2+0x472) [0x55c0eaa6a372] ->
> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c0eaa6a497] ->
> doveadm(main+0x1d4) [0x55c0eaa47c54] ->
> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f35c3d34555] ->
> doveadm(+0x1d0ef) [0x55c0eaa480ef]
> Aborted
>
> John
>
After further testing, I believe the fix for doveadm fts lookup command
is ok (at least on a virtual users setup - I don't have system users to
test this with). What I stated above is wrong: it does produce search
results - I missed them in the debugging because the format is a bit
different to doveadm search, but they are there.

So unless someone with system users reports something not working, I'd
recommend the above patch to resolve the mailbox not open
assert/segfault issue.

John