dovecot - Nov 2020 - dovecot fts-solr + solr 8.7.0 upgrade: "Indexing failed: 401 Unauthorized" + "Transaction commit failed: FTS transaction commit failed: backend deinit" ?

On 13/11/2020 19:55, PGNet Dev wrote:
> On 11/13/20 9:56 AM, PGNet Dev wrote:
>>> (2) see here: https://wiki.dovecot.org/Plugins/FTS/Solr
>>>
>>> two useful settings are debug and rawlog_dir=whatever to be added
in
>>> the same line as fts_solr
>
> with
>
> ????fts_solr = ... debug
>
> line #35 @ https://pastebin.com/9ecLQspD _looks_ like the 401's from
> solr server itself.
>
>
> i do NOT currently have
>
> ? service indexer{}
> ? service indexer-worker {}
>
> configured in dovecot conf; i.e., at whatever the defaults are.
> didn't have to before ...
>
> still dunno why the 401. :-/
So I just did a quick check of running dovecot with a standalone
solr-8.7.0 instance and I'm not seeing any issues.

I confirm I haven't configured anything for indexer or indexer-worker in
dovecot, just left the defaults.

For 401's returned from your solr server you'll need to look into how
you set up authentication.

John

On 11/13/20 11:37 AM, John Fawcett wrote:
>> still dunno why the 401. :-/
> 
> So I just did a quick check of running dovecot with a standalone
> solr-8.7.0 instance and I'm not seeing any issues.
+1
> I confirm I haven't configured anything for indexer or indexer-worker
in
> dovecot, just left the defaults.
+1
> For 401's returned from your solr server you'll need to look into
how
> you set up authentication.
FOUND & *avoided* the silly problem.  yay.  TBD what a 'fix' looks
like.

the issue's a mis-handled, URL-encoded "@" in my basicAuth
'pass' string, as used in fts_solr
my security config included

	solr/data/security.json
		{
			"authentication":{
				"blockUnknown": true,
				"class":"solr.BasicAuthPlugin",
				"credentials":{
					"myuser":"my at pass"
				},
				"realm":"My REALM",
				"forwardCredentials": false
			},
			"authorization":{
				"class":"solr.RuleBasedAuthorizationPlugin",
				"permissions":[{
					"name":"security-edit",
					"role":"admin"
				}],
				"user-role":{
					"solr":"admin"
				}
			}
		}

and

	/etc/default/solr.in.sh
		...
		SOLR_AUTH_TYPE=basic
		SOLR_AUTHENTICATION_OPTS="-Dbasicauth=myuser:my at pass"

and

	dovecot/conf.d/10-master.cf

		fts_solr =
url=https://"myuser":"my%40pass"@solr.example.com:8984/solr/dovecot/
use_libfts soft_commit=yes batch_size=250

changing all

-	my at pass
+	my_pass

so that

		fts_solr =
url=https://"myuser":"my_pass"@solr.example.com:8984/solr/dovecot/
use_libfts soft_commit=yes batch_size=250

does the trick.  fts/solr is indexing, solr's logging the activity, and
there appear to be no more perms/auth/access errors.

On 13/11/2020 21:32, PGNet Dev wrote:
> On 11/13/20 11:37 AM, John Fawcett wrote:
>>> still dunno why the 401. :-/
>>
>> So I just did a quick check of running dovecot with a standalone
>> solr-8.7.0 instance and I'm not seeing any issues.
>
> +1
>
>> I confirm I haven't configured anything for indexer or
indexer-worker in
>> dovecot, just left the defaults.
>
> +1
>
>> For 401's returned from your solr server you'll need to look
into how
>> you set up authentication.
>
> FOUND & *avoided* the silly problem.? yay.? TBD what a 'fix'
looks like.
>
> the issue's a mis-handled, URL-encoded "@" in my basicAuth
'pass'
> string, as used in fts_solr >
> my security config included
>
> ????solr/data/security.json
> ??????? {
> ??????????? "authentication":{
> ??????????????? "blockUnknown": true,
> ??????????????? "class":"solr.BasicAuthPlugin",
> ??????????????? "credentials":{
> ??????????????????? "myuser":"my at pass"
> ??????????????? },
> ??????????????? "realm":"My REALM",
> ??????????????? "forwardCredentials": false
> ??????????? },
> ??????????? "authorization":{
> ???????????????
"class":"solr.RuleBasedAuthorizationPlugin",
> ??????????????? "permissions":[{
> ??????????????????? "name":"security-edit",
> ??????????????????? "role":"admin"
> ??????????????? }],
> ??????????????? "user-role":{
> ??????????????????? "solr":"admin"
> ??????????????? }
> ??????????? }
> ??????? }
>
> and
>
> ????/etc/default/solr.in.sh
> ??????? ...
> ??????? SOLR_AUTH_TYPE=basic
> ??????? SOLR_AUTHENTICATION_OPTS="-Dbasicauth=myuser:my at pass"
>
> and
>
> ????dovecot/conf.d/10-master.cf
>
> ??????? fts_solr >
url=https://"myuser":"my%40pass"@solr.example.com:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
>
> changing all
>
> -??? my at pass
> +??? my_pass
>
> so that
>
> ??????? fts_solr >
url=https://"myuser":"my_pass"@solr.example.com:8984/solr/dovecot/
> use_libfts soft_commit=yes batch_size=250
>
> does the trick.? fts/solr is indexing, solr's logging the activity,
> and there appear to be no more perms/auth/access errors.
I guess you didn't need to enclose username and password in quotes, i.e.

fts_solr url=https://myuser:my%40pass at solr.example.com:8984/solr/dovecot/
use_libfts soft_commit=yes batch_size=250

John