Xapian discuss - Jul 2009 - FW: Wondering about the n-gram QP ?

Apologies, silly corporate auto-boilerplate.

Thanks for letting me know the good news Richard.

We had attempted to create a Perl module which has the same interface as
Xapian::Search::Stem (i.e. new() and stem_word()) and then say
$qp->set_stemmer(new PurePerlNGramStemmer());

However we were thwarted by this error message
"Search::Xapian::QueryParser::set_stemmer() -- stemmer is not a blessed SV
reference".  Might there be a way we could trick QueryParser into using a
pure Perl module, rather than an opaque C++ object ?

tom



>
>  ------------------------------
> *From:* boulton.rj at googlemail.com [mailto:boulton.rj at googlemail.com]
*On
> Behalf Of *Richard Boulton
> *Sent:* 21 July 2009 22:07
> *To:* Thomas Leitch
> *Cc:* xapian-discuss at lists.xapian.org
> *Subject:* Re: [Xapian-discuss] Wondering about the n-gram QP ?
>
>  2009/7/21 Thomas Leitch <Thomas.Leitch at bbc.co.uk>
>
>> I know a good while ago some code was floating around to create an
>> n-gram query parser. I'm just wondering if there are any current
plans
>> for something like that to find its way into Xapian proper ?  We're
>> currently pre-generating a load of grams before pushing them into a
>> Xapian database, but increasingly we're starting to use custom
boolean
>> prefixes.  Having a native n-gram query parser would be fab.
>>
>
> There aren't any definite plans that I know of, but it's certainly
> something we'd like to do - just haven't got around to it yet.
>
> The relevant ticket is http://trac.xapian.org/ticket/180 - the ticket is
> currently marked as something to address in the 1.1 series, and has a high
> priority, so the chances are reasonable that it'll get done in the next
few
> months.
>
> Patches (particularly if they include tests!) will help this process along,
> obviously!
>
>
> This e-mail (and any attachments) is confidential
>
>
> No it's not - you've posted it to a public, and publically
archived,
> mailing list.
>
>
>> If you have received it in error, please delete it from your system.
>> Do not use, copy or disclose the information in any way nor act in
>> reliance on it and notify the sender immediately.
>> Please note that the BBC monitors e-mails sent or received.
>> Further communication will signify your consent to this.
>
>
> I don't believe I've received this message in error, so I won't
delete it.
>  Seriously though, these unreasonable signatures are a bit of a barrier to
> communication; I know that some developers on this list will take them at
> their word, delete the message, and not reply to it.  So, please don't
send
> messages with them.  If your corporate IT policies require them to be
> automatically added, I suggest using a free email address, such as a
> googlemail address, which doesn't have such restrictions.
>
> --
> Richard
>

On Wed, Jul 22, 2009 at 12:03:37PM +0100, Tom Leitch
wrote:
> We had attempted to create a Perl module which has the same interface as
> Xapian::Search::Stem (i.e. new() and stem_word()) and then say
> $qp->set_stemmer(new PurePerlNGramStemmer());
> 
> However we were thwarted by this error message
> "Search::Xapian::QueryParser::set_stemmer() -- stemmer is not a
blessed SV
> reference".  Might there be a way we could trick QueryParser into
using a
> pure Perl module, rather than an opaque C++ object ?
No, the C++ class system doesn't support "duck typing", so that
approach
is just not going to fly.

You can't currently subclass from Xapian::Stem either.

But how would this work anyway?  Xapian::Stem::stem_word() doesn't
support returning more than one result, so you can't return all the
n-grams for the word you are passed.

Cheers,
    Olly