Solr

John Tulp
Wed, 02 Jan 2019 09:10:15 -0000

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote:
> > The main problem is : After some time of indexing from Dovecot,
Dovecot
> > returns errors (invalid SID, etc...) and Solr return "out of
range
> > indexes" errors
>
> I've been watching the progress of this thread with no small concern,
mainly
> because I've been tasked with providing a server-side email search
facility
> with a budget and manpower level that comes down to mainly *1*, i.e., me.
>
> I was expecting, given the strongly worded language about "just use
> lucene/SOLR" and "ignore squat", that I should invest time +
effort into this
> JAVA nightmare that is SOLR.
>
> I started with squat and another word-indexor system that used out-of-band
> (not a dovecot plugin) software to provide rapid (sub-second) searches
through
> tens-of-GB-scale mailboxes.
>
> Unlike what I was led to believe, the squat indexes worked surprisingly
well,
> once you sorted out the odd resource size (ulimit-related) issues (vsz
&
> friends) limitations. I did notice the "worst-case" search
performance have
> worryingly high O(x) increases in time, but I'd not seen anything that
was a
> dealbreaker. It goes without saying that various substring searches worked
as
> expected, for the most part.
>
> My experiences with SOLR were similar to Messr. Moreau's: lots of
startup
> errors with provided schemata files. Lots of JAVA nonsense issues. Lots of
> sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific
JVM,
> version of SOLR, and dovecot to find the "best" working
combination, only to
> find that the searches didn't work out as expected. I expected to be
able to
> do date-ranging based searches. Didn't work. I expected to search
CONTENTS of
> emails, and despite many days of tweaks, I couldn't get it to index
even the
> basics like filenames/types of attachments, so I could exposed
> attachment-based searching to my users.
>
> So, without rancour or antipathy, I ask the entire list: has ANYONE gotten
a
> Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of
the
> following functionality:
>
> 1) The ability to search for a string within any of the structured fields
> (from/subject) that returns correct results?
>
> 2) The ability to search for any string within the BODY of emails,
including
> the MIME attachment boundaries?
>
> 3) The ability to do "ranging" searches for structures within
emails that
> decompose to "dates" or other simple-numeric data?
>
> OPTIONALLY, and this is probably way outside of the scope of the above,
> despite the fact that it's listed as a "selling point" of
SOLR versus other
> full text search engines:
>
> 4) The ability to do searches against any attachments that are able to be
> post-processed and hyper-indexed by SOLR+Tika?
>
> -------------
>
> SOLR seems to have "brand cachet", so presumably it actually
works (for somebody).
>
> Dovecot has not a little "brand cachet", and for me, I have
innate faith and
> trust in Timo and his software. I am no stranger to the "costs"
of "free"
> software, in that you sacrifice your own blood, sweat, and tears just to
get
> these disparate pieces to work together.
>
> I *DO* respect that Timo has to keep the lights (and sauna) on in Finland.
> Maybe there's a super-secret (no advertised prices,
"carrier-only" price list)
> with _Dovecot, Oy_ wherein the above ARE actually available for something
less
> than 6.022 x 10^23 Euros per centi-second of licencing fees.
>
> But please, level with us faithful users.  Does this morass of Java B.S.
> actually work, and if not, please just deprecate and remove this moribund
> software, and stop trying to bury the only FTS plugin many of us HAVE
actually
> gotten to work.  (Pretty please?)
>
> I respect that Messr. Moreau has made an earnest effort to get this JAVA
B.S.
> to actually work, as I have.
>
> He persevered where I'd given up. He's vocal about it, and now
I'm chiming in
> that this ornate collection of switchblades only cuts those who try to use
them.
>
> Respectfully,
> =M>
Fascinating...

SOLR says the following are powered by SOLR...

https://wiki.apache.org/solr/PublicServers

Perhaps if you could find out from that list which of them are using
SOLR in conjunction with Dovecot...

food for thought...