Xapian discuss - Apr 2007 - New to Xapian (coming from Lucene)

Hello all. As the subject says, i am new to Xapian and would like to replace
Lucene with it (so far). The main reason, in case you are curious, is
because we are using this with Perl, and are unhappy with Plucene's handling
of file locking.

As i continue to read through vairous Xapian docs and code examples, i have
noticed that Xapian appears to want positional parameters to store values
into a document. For example, we are indexing book information, such as
TITLE, ISBN, AUTHORS, etc. With Lucene, one can add these attributes to a
document by specifying arbitrary key names:

doc->add_value('TITLE', 'How To Add Searching To Your Site')

However, XAPIAN appears to only accept numbers, and not keys that could be
used to retrieve a value.

doc->add_value(42, 'So Long And Thanks For All The Fish')

Is this really the case? If so, that is fine, we can always define constants
in our code.

But how do now search for, say ISBN numbers?

Thank you in advance for any answers, suggetsions, and insight.

-- 
jeffa

On Mon, Apr 09, 2007 at 01:18:39PM -0400, Jeff Anderson wrote:
> As i continue to read through vairous Xapian docs and code examples, i have
> noticed that Xapian appears to want positional parameters to store values
> into a document. For example, we are indexing book information, such as
> TITLE, ISBN, AUTHORS, etc. With Lucene, one can add these attributes to a
> document by specifying arbitrary key names:
> 
> doc->add_value('TITLE', 'How To Add Searching To Your
Site')
> 
> However, XAPIAN appears to only accept numbers, and not keys that could be
> used to retrieve a value.
Hi, Jeff. I think you're running into (understandable) confusion here
because of differences in naming conventions between Xapian and
Lucene.

Xapian has two *main* things you worry about putting into your
document:

 * TERMS (which are used for searching)
 * DATA (which is most often used for displaying the search results)

Values are quite a specialised thing, so don't worry about them for
now.

Terms come in two forms: postings (which have positional information)
and "plain" terms (which don't). So you can do:

----------------------------------------------------------------------
doc->add_posting("term", position);
doc->add_term("anotherterm");
----------------------------------------------------------------------

for instance. Neither term not anotherterm will be stemmed or split
apart at this point - this is real low level stuff.

The document data is simply a chunk of text or binary data which is
stored alongside the document. It is most commonly used to implement
what we call FIELDS as name-value pairs, by doing something like:

----------------------------------------------------------------------
title=How To Add Searching To Your Site
authors=Jeff Anderson
----------------------------------------------------------------------

Xapian doesn't provide any support itself for doing this, but in Perl
it's pretty easy to do (you could, in fact, serialise a Perl hash to
make life really easy).

Ask if any of that doesn't make sense.


Now. Many of your fields are going to be things you also want to
search on, such as title and author. What you probably want to do is
to create terms out of those field values, as well as the fields
themselves. This is where we get into term prefixes. Have a read
through omega/docs/termprefixes.txt and come back with any questions
:)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james@tartarus.org                               uncertaintydivision.org

> Message: 2
> Date: Thu, 12 Apr 2007 12:15:59 +0100
> From: James Aylett <james-xapian@tartarus.org>
> Hi, Jeff. I think you're running into (understandable) confusion here
> because of differences in naming conventions between Xapian and
> Lucene.
Perhaps that was because the Lucene docs were more accessible than the
Xapian docs. Sorry to sound mean, but the fact is i found Lucene to be
much easier to understand and to actually use.

> Terms come in two forms: postings (which have positional information)
> and "plain" terms (which don't). So you can do:
I still don't understand the need to specifiy a numeric position,
unless this determines some kind of "boost" on the term. In Lucene and
Kinosearch, the coder can refer to items by a key name, and be able to
retrieve pieces of data to display in the search results by those key
names. Leave the numbers to the mathematicians! :P

> The document data is simply a chunk of text or binary data which is
> stored alongside the document. It is most commonly used to implement
> what we call FIELDS as name-value pairs ... Xapian doesn't provide
> any support itself for doing this, but in Perl it's pretty easy to do
(you
> could, in fact, serialise a Perl hash to make life really easy).
And this is what really gets to me. I just don't agree with this
philosphy, as i have already stated a few times previously. As i also
stated earlier that i decided to switch to Kinosearch. No offense, but
it's just a better fit for my needs.