Xapian discuss - Sep 2009 - Strange Weighting issue

Hello,

Our client has asked us for the ability to boost documents on a
document type basis. So we have a scale or 1-5 for the client to pick
from and in the background we had a sort of logarithmic scale between
1 and 100.

I am seeing very strange results however.

I have about 450 documents, I have set all of them to a weight of 1 like so:

$indexer->index_text($this->getTitle(), 1);
$indexer->index_text($this->getDescription(), 1);
$indexer->index_text($contenttext, 1);

One document I set to a weight of 100 like so:

$indexer->index_text($this->getTitle(), 100);
$indexer->index_text($this->getDescription(), 100);
$indexer->index_text($contenttext, 100);

If I search using the two words that is in the title of the document
with a weight of 100, the document with the 100 weight returns last.

If I re-index every document with a weight of 0 and leave the other
document as a weight of 100 it appears as the top document.

If I re-index every document leaving the weight to default and leave
the other document as 100 the 100 document appears at the bottom.

If I re-index every document with a weight of 1000 and leave the other
document as a weight of 100 it appears as the second top document!

I do a reindex with everything as 10000 and the 100 document as 10 the
100 document returns in position 3.

As a base line I have reindexed with no weighting on any document and
the document I am trying to give a boost to appears on page 2
(position 11 or rank 10 in xapian speak).

Am I doing the weighting wrong? I am really confused about this.

Thanks
John

2009/9/8 John Wards <jwards at whiteoctober.co.uk>
> Our client has asked us for the ability to boost documents on a
> document type basis. So we have a scale or 1-5 for the client to pick
> from and in the background we had a sort of logarithmic scale between
> 1 and 100.
>
(I assume that $indexer in your code is an instance of
Xapian::TermGenerator)

The issue here is that the second parameter to TermGenerator, although it is
called "weight", is actually an multiplier applied to the frequency of
the
term within the document.  In other words, setting the weight in this
situation to 100 is equivalent to repeating the title 100 times (except that
the positional information generated will be different).  This increases the
frequency of the term within the document, but also increases the length of
the document.

The default Xapian weighting formula has a term which reduces the weight of
large documents; the theory being that a small document with relevant
information in it is better than a large one, because the small one is
likely to be more tightly focussed on the topic.  As you're seeing, this
compensation can mean that the same document repeated 100 times is
considered less good than a single repetition of the document.

While the "weight" parameter you're using can be useful for giving
particular fields a bonus weight compared to others, in general it is too
blunt an instrument to be used to reflect document-wide weights.

Instead, if you're using the 1.0 release series, I recommend using a single
extra term, added to all the documents, but with a weight chosen according
to the document's importance.  Using a single term shouldn't disrupt the
document length so much, but should allow you to weight your searches
appropriately (you'll need to include that term in all your queries, of
course: combine your existing queries with the weight term using the
AND_MAYBE operator).

If you're using the 1.1 development series, you can store an arbitrary
numeric weight for each document, and use a ValueWeightPostingSource to add
this weight to the standard weight of the document.  Using this technique,
you can specify exactly what weight bonus you want to be added to each
document.

If I re-index every document with a weight of 0 and leave the
other
> document as a weight of 100 it appears as the top document.
>
This makes sense - the documents indexed with a weight of 0 will be given a
total weight of zero, so the one with 100 weight will come top.

-- 
Richard