Xapian discuss - Feb 2006 - best method for stemming

Hi,

    I am implementing an index using xapian via the perl bindings and
I'd like to know what the preferred method is for stemming.  Having
browsed through the archives I've seen several approaches.  I will be
indexing millions of documents so I want to make sure this is done
optimally.  I'm willing to sacrifice disk space for faster lookups. 
Is it better to stem while indexing or to stem the query and treat it
like a wildcard or am I off all together?  Right now as I iterate
through the document, I stem the words and add the stem to the index
at the same position as the non-stemmed word.

Thanks,

Alex

On Tue, Feb 07, 2006 at 05:15:49PM -0500, Alex Deucher
wrote:
> I'm willing to sacrifice disk space for faster lookups.
For a large corpus you're likely to be I/O bound, so a larger database
will tend to mean slower lookup because Xapian is more likely to need
to actually read information from disk rather than find it in the OS
disk cache.
> Is it better to stem while indexing or to stem the query and treat it
> like a wildcard or am I off all together?
Stemming while indexing is a pretty common approach.  At query time you
also stem all query terms.  The plus points are that you have a reduced
number of terms (and so a smaller database).  The main downside is that
you can't search for unstemmed terms, which can be a problem sometimes
especially is a proper name is conflated with a common word.

Alternatively, you could stem nothing at index time and then for search
terms which you want to stem, stem them, and then run them through an
"unstemming" algorithm to produce a list of terms they could have come
from.  Then OR this list together.  Unfortunately nobody has written
the "unstemmer" yet.  Also this means more work at search time than
the first approach, but that may not really matter.  I've not tried
the idea, so I can't say for sure.

Omega's approach is to store everything stemmed and selected terms
unstemmed (those from capitalised words).  Unstemmed terms have an
"R" prefixed.  This allows name searches to work.
> Right now as I iterate through the document, I stem the words and add
> the stem to the index at the same position as the non-stemmed word.
I've not tried this approach, but is seems plausible.  It means more
term postings for R terms than Omega generates, but storing a term with
many postings is more efficient per posting so it may not be so bad.

You can probably get away with not storing positional information for
the stemmed terms (use add_term instead of add_posting).  That means
that NEAR and phrase searches will only work on unstemmed terms, but
that's not a big restriction and will save a lot of diskspace - the
position table is generally the largest.

Note that the QueryParser currently only really supports Omega's
approach, stemming everything, or stemming nothing.  I regard that
as a bug, but some thought is need as to how best to make it
more configurable.

Cheers,
    Olly