Xapian discuss - Aug 2004 - Filter or MatchDecider or Other?

I have about 35,000 documents stored in MySQL. I have a custom-made
PHP-based search tool for searching this data, and it works well, but does
not support boolean or phrase searching. I'd like to use Xapian to provide a
search with these features.

My existing search tool allows me to limit the search results based on other
parameters besides the text content. For example, each document has an
associated category, and I can limit my search to only a certain category.
Similarly, I can limit the search by a geographic region, or by a date
range.

I've built a Xapian-database of just the text content, and the simplesearch
tool seems to return relevant results as expected. What's the best way to
add the limiting to the search?

It looks like I could maybe add a term of something like
'Location:Location1' to the end of my document and then add OP_FILTER to
the
query and require that piece of data. Or perhaps I could set the catgory as
a document value and then implement the MatchDecider to limit on that?

I haven't found the documentation very clear as to which way will work, or
which way is preferred/faster. Please point me in the right direction.

BTW, I'm developing this with the PHP Xapian bindings, please let me know if
a certain feature won't work under this setup.

Thanks!
Mike Boone
http://boonedocks.net/mike

On Thu, Aug 19, 2004 at 06:24:36PM -0400, Mike Boone
wrote:
> My existing search tool allows me to limit the search results based on
other
> parameters besides the text content. For example, each document has an
> associated category, and I can limit my search to only a certain category.
> Similarly, I can limit the search by a geographic region, or by a date
> range.
> 
> I've built a Xapian-database of just the text content, and the
simplesearch
> tool seems to return relevant results as expected. What's the best way
to
> add the limiting to the search?
> 
> It looks like I could maybe add a term of something like
> 'Location:Location1' to the end of my document and then add
OP_FILTER to the
> query and require that piece of data.
That's the best way to filter on pre-defined categories.  Pick a syntax
for the term which won't clash with terms generated from the text
(including a colon like you suggest is fine; the convention Omega uses
is that terms from text are lower case, so capital letter prefixes are
used for filter terms).
> Or perhaps I could set the catgory as a document value and then
> implement the MatchDecider to limit on that?
That's likely to be less efficient than using OP_FILTER.  A document
value is designed to be fast to access, but using a term means the
list of document ids is effectively precalculated.

A MatchDecider is useful when the decision is more complicated - for
example you might want to restrict results to "within X miles of P"
which
would be hard to do efficiently with OP_FILTER, but a MatchDecider could
take the coordinates from a value and calculate the distance from P,
saying "yes" only if that distance is less than X.

For a date range, I'd suggest considering the scheme Omega uses - there
we generate terms for the date (e.g. D20040820), the month (e.g. M200408),
and the year (e.g. Y2004).  This allows a long date range to be
represented as a relatively small number of terms OR-ed together.  If
you just indexed the D terms, a year span would require 365 terms.
> I haven't found the documentation very clear as to which way will work,
or
> which way is preferred/faster. Please point me in the right direction.
Thanks for the feedback - I'll slot the above suggestions into the
documentation in a suitable place.
> BTW, I'm developing this with the PHP Xapian bindings, please let me
know if
> a certain feature won't work under this setup.
I'm not certain if MatchDecider is supported by the PHP bindings.  You
need to be able to sub-class it to use it, or supply the decision
function in some other way.  Hopefully someone with more knowledge of
the bindings than me can give a more definitive answer...

Cheers,
    Olly