Xapian discuss - Jan 2005 - Evaluating Xapian

Hello

We're currently evaluating replacing SWISH-E with Xapian for a couple of
search installations.

The problem we have is, that the searches are no "pure" fulltext
searches, but also need some features which are otherwise more likely to
be found in RDBMS. Due to performance reasons we unfortunately cannot
combine the RDBMS results with the results from the fulltext search
engine, but instead, everything needs to come from the fulltext search
engine.

SWISH so far does, what we need in terms of search features, but its
ranking mechanism is not very good, and live indexing of documents is
only an afterthought there. Therefore we currently evaluate Xapian.

The most complex search application which we want to replace is a kind
of "person" search. If a search engine can fullfill all the
requirements
of that application, it will also work for all our other needs.

A person is defined by age/birthdate, gender, contact information like
state, country, city in different fields, as well as freetext documents
attached to it. It must be possible to freely combine searches for the
different fields. That means that I can for example search for all male
persons between 20 and 30 from the USA where the word "foo" occurs in
the attached freetext documents.

The questions now are:
- Can I, and if yes, how would I do it, restrict searches to a specific
age group? SWISH-E offers the "-L" option for such range searches, but
I
have not yet found something similar in the Xapian documentation. Range
search features are also useful for such things like "give me all
documents matching 'foo bar' which have been modified the last 30
days". I really really want to avoid adding a seperate filtering step
afterwards for things like that. Omega seems to implement a feature like
that, but how?
- Does anyone have good Perl-based examples for the indexer and the
searcher as a starting point?
- The documentation I read so far is not very explicit on searching
different fields. The way I currently understood it, I simply make the
name of the fields I want to support part of the terms I add to the
documents?

-- 
Markus Peter - SPiN AG                                          warp@spin.de

On Mon, Jan 24, 2005 at 02:04:09PM +0100, Markus Peter
wrote:
> A person is defined by age/birthdate, gender, contact information like
> state, country, city in different fields, as well as freetext documents
> attached to it. It must be possible to freely combine searches for the
> different fields. That means that I can for example search for all male
> persons between 20 and 30 from the USA where the word "foo"
occurs in
> the attached freetext documents.
> 
> The questions now are:
> - Can I, and if yes, how would I do it, restrict searches to a specific
> age group?
You can form a Query object as an OP_OR of all the filter terms, then
apply it to your probabilistic query with OP_FILTER.  In perl, that
looks something like (untested):

    my $query = Search::Xapian::Query->new("foo");
    my $filter = Search::Xapian::Query->new(OP_OR, map {"XAGE$_"}
(20..30));
    $filter = Search::Xapian::Query->new(OP_AND, $filter, "XSEXm");
    my $enquire = $db->enquire(OP_FILTER, $query, $filter);

This assumes a document is indexed by the term XAGE20 if the person is 20
years old, and by XSEXm if the person is male.
> Range search features are also useful for such things like "give me
all
> documents matching 'foo bar' which have been modified the last 30
> days". I really really want to avoid adding a seperate filtering step
> afterwards for things like that. Omega seems to implement a feature like
> that, but how?
Omega does something slightly more complicated for date ranges to avoid
OR-ing really large lists of terms.  Instead there are terms
representing whole months, and whole years, and these are used whenever
a range includes whole months and/or years, with the ends of the range
being made up of terms covering a single date, if necessary.

If you are creating very large ranges, you might want to experiment
with a similar approach.
> - Does anyone have good Perl-based examples for the indexer and the
> searcher as a starting point?
For searching, see:

Search-Xapian-0.8.4.0/examples/simplesearch.pl

I don't have a good example perl indexer to hand, though the API is
pretty similar to C++, so you can probably use the C++ example.
> - The documentation I read so far is not very explicit on searching
> different fields. The way I currently understood it, I simply make the
> name of the fields I want to support part of the terms I add to the
> documents?
Pretty much.  Conventionally, the prefix you add should be all uppercase.
Multi-letter prefixes should start with "X", and for such prefixes
you need to add a ":" to separate prefix and term if the term itself
starts with a capital letter.

Single letter prefixes (i.e. A-W, Y, and Z) are reserved for standard
uses (e.g. D is a date in the format YYYYMMDD - today would be
D20050124) so that applications like Omega know what to do with them,
and so that you can combine searches over arbitrary database.

But if you don't care about compatibility, you can mostly ignore these
conventions.  In the main library, only the queryparser cares about
them, and only in that it needs to know when to add a ":", and that
it'll use the R prefix for "raw" terms.

At some point, we intend to make the prefix and term combining more
transparent.  Once you know the rules, the current system works very
well, but it looks a bit quirky.

Cheers,
    Olly