Xapian discuss - Oct 2007 - indexing of individual fields + all fields

In our search app, users currently search 4 different fields at the same 
time.

I am adding functionality so that users can optionally limit their 
search to individual fields.

I was able to get this to work with omega scriptindex by indexing these 
fields twice... once as a group, and once individually:

   field1 : weight=4 index index=XFIELD1
   field2 : weight=3 index index=XFIELD2
   field3 : weight=2 index index=XFIELD3
   field4 : weight=1 index index=XFIELD

I have tested this and it seems to work, but I am wondering if it is the 
best way to implement it.

The solution seems like it would provide the best performance for 
searches, since most of the searches will search all fields at once, but 
probably take longer to build the index and more storage space to store 
the index.

Thoughts?

- Jim

On Mon, Oct 29, 2007 at 01:34:07PM -0400, Jim Spath
wrote:
> In our search app, users currently search 4 different fields at the same 
> time.
> 
> I am adding functionality so that users can optionally limit their 
> search to individual fields.
> 
> I was able to get this to work with omega scriptindex by indexing these 
> fields twice... once as a group, and once individually:
> 
>   field1 : weight=4 index index=XFIELD1
>   field2 : weight=3 index index=XFIELD2
>   field3 : weight=2 index index=XFIELD3
>   field4 : weight=1 index index=XFIELD
> 
> I have tested this and it seems to work, but I am wondering if it is the 
> best way to implement it.
In current releases (<= 1.0.3), this approach is the only option.  1.0.4
(out later this week) will add support to the QueryParser for searching
for a term with any of several prefixes (so you could just index the
fields with a prefix).
> The solution seems like it would provide the best performance for 
> searches, since most of the searches will search all fields at once, but 
> probably take longer to build the index and more storage space to store 
> the index.
I don't think anyone has benchmarked the new approach against the old
for a case like this, but I wouldn't be suprised if it's still
worthwhile indexing as above if most searches want to search all
fields.

If you try both approaches, it would be interesting to hear how they
compare.

Cheers,
    Olly