Xapian discuss - Oct 2005 - Stemmer Modifications

I'm using Xapian as a search back-end on a website. My client has
certain search terms that the stemmer does not stem in a way they would
like. For example "continuity" stems to "continu", which
produces
undesirable results in their application. Is there a way to override the
stemming of certain words in a way that is compatible with the indexing
stemmer and the query parser stemmer?

Thanks,
Mike.

On Tue, Oct 11, 2005 at 09:52:08PM -0400, Mike Boone
wrote:
> I'm using Xapian as a search back-end on a website. My client has
> certain search terms that the stemmer does not stem in a way they would
> like. For example "continuity" stems to "continu",
which produces
> undesirable results in their application. Is there a way to override the
> stemming of certain words in a way that is compatible with the indexing
> stemmer and the query parser stemmer?
Not that I can see without changing the xapian-core code.

If Xapian::Stem::operator() was virtual, you could subclass
Xapian::Stem, overriding operator() and either implementing your own
stemmer, replacing stems for certains terms, or post processing the
results of an existing stemmer.  The current QueryParser API takes
a Xapian::Stem object to specify the stemmer so you could pass in
a subclassed one.

The only real downside I can see is this change would add a tiny
overhead from the virtual message dispatch, which I don't see as
an issue really.

Cheers,
    Olly

On Tue, Oct 25, 2005 at 11:47:08AM -0400, Mike Boone
wrote:
> I'm finally getting back to this topic after some distractions. I was
> reading the Snowball documentation here:
> 
> http://snowball.tartarus.org/algorithms/english/stemmer.html
> 
> Which has a section on "exceptional forms." That looks like what
I want
> to do, but that language isn't familiar. I looked over the source code
> in xapian-core/languages/snowball_english.cc and the stemmer code looks
> pretty cryptic.
That's generated code, which is why it's hard to follow.  The
snowball-to-C convertor doesn't regard readability of the generated code
as important!
> Is it possible to put my exception word list into that code?
It's probably workable as a local patch, but it's not something we could
really contemplate including in the mainstream releases.  Actually, you
might find it easier to tap into Xapian::Stem::operator() in
api/omstem.cc - that's human written code and is where we call the
Snowball generated code from.

Or you could modify the Snowball stemmer code, then regenerate the C
code in snowball_english.cc (it's really just C, but we compile it as
C++ because I found it causes fewer headaches).  The Snowball language
takes a little bit of getting used to, but is fairly elegant once you
get the hang of it.
> That seems clearer in my mind than subclassing the Stem class and
> then exposing that subclass to PHP through the SWIG bindings (maybe
> that's easier than it sounds?).
I don't think SWIG's PHP bindings support what they call
"directors" yet
so no, you probably can't currently subclass in PHP.  Someone's recently
picked up the reins as SWIG PHP maintainer, so it's possible this might be
implemented sometime (assuming PHP permits it).

Cheers,
    Olly