Xapian discuss - Jun 2005 - scriptindex on an internet crawl

Hi!

I have a crawl of the chilean internet on disk in raw text. I have a
field for the url of the document, for the title and for the content.

As a first try, I used scriptindex with

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
uid : field=uid boolean=XUID unique=XUID
title : field=title index
value : index
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

in the command file and, as input file

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
uid=1
title=us spy plane crashes in sw asia
value=a us air force u-2 spy plane has crashed in south-west asia
killing
=the pilot, the us military has said.  the crash occurred at 2330 gmt
=on tuesday, when the pilot was returning to base after completing a
etc.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

and it works fine for probabilistic searches.

My questions are:
1) what should I input in the search field to search only the title? I
was expecting that something like "title: plane" would work, but it
doesn't.
2) how should I do to see the original url of the documents retrieved
such that if I click on the hyperlink, I am redirected to the original
document (i.e. not the document I have in my copy of the crawl). In a
first try, I inserted in the command file url : field=url boolean=XURL
unique=XURL and in the input file: url=www.dcc.uchile.cl/~gdupret for
example, but scriptindex start using 100% of the CPU and never finishes.

Thank you in advance for your help

Georges

On Wed, Jun 22, 2005 at 03:21:32PM -0400, Georges Dupret
wrote:
> 1) what should I input in the search field to search only the title? I
> was expecting that something like "title: plane" would work, but
it
> doesn't.
You need to index the title with a suitable prefix for this to work.
Make the index script line for the title:

    title : field=title index=S index

And then in your omegascript template (i.e. templates/query), add this
(at the start is best):

    $setmap{title,S}

Omega's docs/termprefixes.txt and the $setmap section of
docs/omegascript.txt explain this, but could do with being less
disjoint.  I'll fix that.
> 2) how should I do to see the original url of the documents retrieved
> such that if I click on the hyperlink, I am redirected to the original
> document (i.e. not the document I have in my copy of the crawl).
Assuming you want to store the original url in the database (which
probably is the best approach) then you'll need to modify your
omegascript template.  Look for where it uses $field{url} and modify
that so that the link given is for the locally cached copy.
> In a first try, I inserted in the command file url : field=url boolean=XURL
> unique=XURL and in the input file: url=www.dcc.uchile.cl/~gdupret for
> example, but scriptindex start using 100% of the CPU and never finishes.
You probably don't want to specify both uid and url as unique fields,
but this should cause a hang - I'll see if I can reproduce this.  Just
this rule should do:

    url : field=url

Cheers,
    Olly