Xapian discuss - Apr 2006 - About field weight

Hi,

I've been disappointed today with some of my Xapian results.
Here is the issue :

- I am searching for the terms "ipod vid?o 60" (with the OR operator)
- the first results sorted by relevance are :  (name / description)

   1. Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Blanc

   Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Noir -
   With the SA1 adapter and inMotion, listen to your favorite tunes on any
   inMotion speakers.

   2. Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Noir

   --

   3. Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Rouge

   --

   4. Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Violet

   --

   5. L'iPod vid?o 60 Go

   Comblez vos yeux et vos oreilles ! Musiques, photos, podcasts, et
   maintenant vid?o
 la r?volution se poursuit. Avec jusqu'? 15000 chansons et
   pochettes d'albums en couleur , c'est l'int?gralit? de votre
discoth?que que
   vous pourrez embarquer. Grande nouveaut?, vous stockerez jusqu'? 150
heures
   de vid?o au format MPEG-4. Vous aurez en plus un grand confort
d'utilisation
   gr?ce ? l'?cran de 2,5 pouces (6,3 cm). Quant aux photos, ce sont pas
moins
   de 25000 images au format jpeg que vous emm?nerez partout pour les regarder
   et les montrer quand vous le voudrez. Revivez vos meilleurs souvenirs gr?ce
   aux diaporamas sur fond musical. En terme de design, il est encore plus
   ?l?gant : avec moins d'un centim?tre et demi d'?paisseur , il ne vous
   quittera plus. Plus de performances dans moins d'espace, c'est cela
le
   nouvel iPod. Visionnez des podcasts vid?o et vos clips pr?f?r?s. Consultez
   votre agenda ou recherchez une adresse e-mail
 Gr?ce ? la molette sensitive
   , vous aurez acc?s ? l'int?gralit? de votre discoth?que, photot?que ou
   vid?oth?que tr?s simplement et tr?s rapidement. Sur iTunes Music Store ,
   vous aurez le choix parmi plus de 2 millions de chansons, 25000 podcasts,
   1000 clips vid?os. T?l?chargez-les sur votre Mac ou votre PC, puis
   synchronisez-les sur votre iPod. Autonomie jusque 20 heures. Ecran TFT 320 x
   240 pixels. Dim. 10,3 x 6,1 x 1,4 cm.

- I don't understand howcome the 6th results is not in first position.
Indeed, here is a part of my scriptindex configuration :
name: weight=5 unhtml index index=S field=caption
description: weight=1 unhtml index field=sample

that means *name* should be considered 5 times more relevant than
*description*, isn't it ?

Maybe I should have put higher weight to the *name* field ? like 20 or 100
instead of 5 ?

Thanks in advance !

--
David LEVY {selenium}
Website ~ http://www.davidlevy.org
Wishlist Zlio ~ http://david.zlio.com/wishlist
Blog ~ http://selenium.blogspot.com

On Sun, Apr 02, 2006 at 05:46:05PM +0200, David Levy
wrote:
> - I am searching for the terms "ipod vid?o 60" (with the OR
operator)
> - the first results sorted by relevance are :  (name / description)
> 
>    1. Etui en cuir Shinnorie EZgoing pour iPod avec vid?o 60 Go - Blanc
> 
>[...]
> 
>    5. L'iPod vid?o 60 Go
>[...]
> - I don't understand howcome the 6th results is not in first position.
I assume you mean 5th not 6th, because you didn't show us the 6th
result...
> Indeed, here is a part of my scriptindex configuration :
> name: weight=5 unhtml index index=S field=caption
> description: weight=1 unhtml index field=sample
> 
> that means *name* should be considered 5 times more relevant than
> *description*, isn't it ?
"5 times more relevant" seems a little misleading.  Specifying
"weight=5" means precisely that an appearance of a term in
"name" is
equivalent to 5 appearances of that term in "description".

But note that, both hits 1 and 5 have all 3 search terms appearing once
in the "name" field.

The overall weight takes the document length into account too (which is
the sum of the wdfs of all terms which index the document).  I assume
the 1st match ranks higher because it has more occurences of the search
terms in "description" given the document length than the 5th.

Xapian doesn't allow you to consider the lengths of individual fields
separately (at least not at present).
> Maybe I should have put higher weight to the *name* field ? like 20 or 100
> instead of 5 ?
Well, both "name" fields will still have all 3 search terms appearing
once.  There would be some effect from a long "name" field inflating
the
document length.

I'd be concerned that using a large weight might decrease the quality of
retrieval results in general though.

Actually, what would give you the result your after is an idea I've had
in the back of my mind for a while, but haven't had a chance to try out
- for want of a better name, I'll call it "auto phrase search".

With this, a query for `ipod video 60' would be roughly equivalent to:

    "ipod video 60" OR (ipod video 60)

So if the phrase matches in a particular document, that document gets a
higher weight than if it doesn't.  If the phrase doesn't match any
documents, the results will be the same as for the query as currently
handled.

Cheers,
    Olly