Xapian discuss - Mar 2006 - Getting started with Omega

Hi all,

We are looking at replacing our existing search engine (htdig) with
Xapian/Omega. I've been reading the mailing list and docs, and it seems
that the best way to do this is to continue with htdig to crawl our
site, (it is spread across many servers), and then import the htdig
database files into xapian.

I think I have managed to import a htdig catalogue, the output from
scriptindex is 
records (added, replaced, deleted) = (1695, 0, 0)

and when I call the cgi it states 
Searching 1,695 documents

However, I can't get any results returned.  I'm not 100% certain that
there are really 1,695 documents, the record_DB and termlist_DB files
are only 40k each.  I have had to edit htdig2omega to make it work as
we're running 3.1.6, but I am not sure it really is working.  Are any of
the database files in human readable form?

Thanks in advance,
Georgina.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Georgina Allbrook    
WebTeam : ITS Division : University of Waikato
http://phonebook.waikato.ac.nz/dept/WWTE.shtml
Ph 64 7 856 2889 x 6086

"Those who will play with cats must expect to be scratched." 
-- Cervantes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mon, Mar 27, 2006 at 05:42:50PM +1200, Georgina Allbrook
wrote:
> However, I can't get any results returned.  I'm not 100% certain
that
> there are really 1,695 documents, the record_DB and termlist_DB files
> are only 40k each.
It sounds like you do have 1695 documents, but they may not contain
very much...
> I have had to edit htdig2omega to make it work as
> we're running 3.1.6, but I am not sure it really is working.
What changes did you make?  It's hard for me to suggest what might be
wrong with a script with unspecified changes...
> Are any of the database files in human readable form?
No, although you can inspect the contents with the delve utility (in the
examples directory of xapian-core).

But in this case it's probably more informative to look at the output
from htdig2omega.  Can you send a small sample?

Cheers,
    Olly

Hi Olly, 

Thanks for the response, delve -v returns:
number of documents = 1695
average document length = 0
number of unique terms = 0

So it would seem that I have 1695 but they are all empty.
My modifications to htdig2omega were quite small.  I updated the system
call to include the full path to htdump and to add an additional switch
-c /path to config/db.conf

However, in thinking about the issue it occurred to me that the command
to call htdig2omega might need a full path to htdig2omega.script.
Adding this in seems to work and now delve shows that the documents do
contain information.

I have since attempted to run htdig2omega against a 3.2b6 database and
am getting the following error
Caught exception in UNIQUE!
E: Key too long: length was 258 bytes, maximum length of a key is
BTREE_MAX_KEY_LEN bytes
Exception: Key too long: length was 258 bytes, maximum length of a key
is BTREE_MAX_KEY_LEN bytes

Is it possible or easy to increase BTREE_MAX_KEY_LEN?

Georgina.

-----Original Message-----
From: Olly Betts [mailto:olly@survex.com] 
Sent: Friday, 31 March 2006 12:38 a.m.
To: Georgina Allbrook
Cc: xapian-discuss@lists.xapian.org
Subject: Re: [Xapian-discuss] Getting started with Omega

On Mon, Mar 27, 2006 at 05:42:50PM +1200, Georgina Allbrook
wrote:
> However, I can't get any results returned.  I'm not 100% certain
that
> there are really 1,695 documents, the record_DB and termlist_DB files
> are only 40k each.
It sounds like you do have 1695 documents, but they may not contain
very much...
> I have had to edit htdig2omega to make it work as
> we're running 3.1.6, but I am not sure it really is working.
What changes did you make?  It's hard for me to suggest what might be
wrong with a script with unspecified changes...
> Are any of the database files in human readable form?
No, although you can inspect the contents with the delve utility (in the
examples directory of xapian-core).

But in this case it's probably more informative to look at the output
from htdig2omega.  Can you send a small sample?

Cheers,
    Olly