dovecot - Apr 2022 - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

Dear Shawn Heisey,

My Dovecot, Postfix, Roundcube and Squirrelmail were automatically
installed, configured and deployed by Virtualmin web hosting control
panel install scripts.

How do I integrate Apache SOLR searching capability with Dovecot? Are
there any excellent guides for this?

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
17 Apr 2022 Sunday


On Sun, 17 Apr 2022 at 15:46, Shawn Heisey <elyograg at elyograg.org>
wrote:
>
> On 4/16/2022 11:32 PM, Turritopsis Dohrnii Teo En Ming wrote:
> > Both Roundcube and Squirrelmail webmail cannot search for anything
> > because there are more than 200,000 emails in my Inbox.
>
> <snip>
>
> > Both Roundcube and Squirrelmail webmail cannot open many emails
> > because there are more than 200,000 emails in my inbox.
>
> <snip>
>
> > My IMAP server is Dovecot version 2.2.36.
>
> I artificially created a situation like this in my mailbox.  I created a
> new folder and copied all my other messages in my mailbox to it in the
> Linux commandline.  Ended up with 154K messages in that folder.  Then I
> had to force a complete reindex. Which I have a script for.
>
> I'm using Solr for dovecot searching.  If you do something similar and
> have dovecot connect to something external for search, that should solve
> the first problem.
>
> The folder did eventually populate in roundcube, though it did take a
> while.  My roundcube version is 1.5.2, and my dovecot version is
> 2:2.3.18-4+ubuntu20.04.  Dovecot is installed from the official project
> APT repository on Ubuntu Server 20.04.
>
> I seem to be having a problem with my Solr search engine.  No matter
> what keyword I search for in that folder, it always gets 127168 hits.
> That is just not possible when searching for "dragon" and then
"aardvark".
>
> Thanks,
> Shawn
>

On 4/17/2022 7:44 AM, Turritopsis Dohrnii Teo En Ming
wrote:
> My Dovecot, Postfix, Roundcube and Squirrelmail were automatically
> installed, configured and deployed by Virtualmin web hosting control
> panel install scripts.
>
> How do I integrate Apache SOLR searching capability with Dovecot? Are
> there any excellent guides for this?
I have this in /etc/dovecot/conf.d/10-mail.conf:

mail_plugins = $mail_plugins quota fts fts_solr

And in conf.d/90-plugin.conf, I have this:

plugin {
 ? fts = solr
 ? fts_solr = url=http://localhost:8983/solr/dovecot/
 ? fts_autoindex = yes
 ? fts_enforced = yes
}

I'm running a version of Solr that I built myself from the branch_8_11 
source code -- 8.11.2-SNAPSHOT.? I installed that with the service 
installer script that it includes.? I have been customizing the index 
config, it started with the config and schema provided by dovecot, which 
did work.? I'm going to have some questions for those who write the 
fts_solr plugin, and I still need to glance at the source code.

I am a committer on the Apache Solr project, so I do know a little bit 
about Solr. :)? I am not as competent in the C code that makes dovecot 
as I am in the Java code that makes Solr.? C was one of my first 
languages, but I have not spent any real time with it in decades.

My install of dovecot has a grand total of 180079 messages in it right 
now, 154K of which are in my mailbox.? (I deleted the test folder I 
created to test your problem out).? If you can upgrade roundcube and 
dovecot, plus incorporate Solr, things might work better for you.

I would suggest that you move most of the messages out of your inbox 
into some kind of archival folder.? Keep between 3 months and one year 
of messages in the actual inbox so working with that folder is fast, and 
you can visit the archive if you need to look at something older.

I have never liked dealing with do-everything packages like webmin or 
virtualmin.? I want to understand each of the pieces that make up the 
whole, and be able to upgrade and reconfigure each piece independently 
from the others.? I find the configuration restraints of such a system 
very limiting.? Very often if you customize something yourself outside 
the GUI, you lose the ability to administer it in the GUI, because it 
doesn't know how to deal with the change.? Or if you do change something 
in the GUI afterwards, your customization might get deleted.

Thanks,
Shawn