Xapian discuss - Jul 2009 - termgenerator + stopper + ruby not working?

Hi,

I'm working on a more Ruby-ish Ruby library for Xapian called
XapianFu[1] but I've run into a problem using the SimpleStopper.

I'm not sure if I'm using it wrong, or if there is a bug with the Ruby
bindings or something so I'm hoping someone can shed some light on it
for me.

I can set up a stopper and it works as expected on its own, but when
used with a TermGenerator it seems not to work.  

I'd expect the TermGenerator to not return terms that the stopper
returned true for, but it does.  This is the expected behaviour right?

I've attached an example Ruby program to reproduce the situation.  Any
clues are much appreciated.

Thanks,

John.
http://johnleach.co.uk

[1] http://github.com/johnl/xapian-fu/tree/master

On Sat, Jul 18, 2009 at 12:33:29PM +0100, John Leach
wrote:
> I'd expect the TermGenerator to not return terms that the stopper
> returned true for, but it does.  This is the expected behaviour right?
Actually, no.  The stopper is used to avoid indexing stemmed forms of
stopwords, but we still index the unstemmed forms so that searches for
phrases containing stopwords can be supported.

This isn't mentioned in the collated API documentation, but is here:

    http://xapian.org/docs/termgenerator.html

I'll add a note to the API documentation.

There probably should be an option to not index stopwords at all, but
there isn't at the moment.

Cheers,
    Olly