Xapian discuss - Jan 2009 - Bug in set_cutoff - xapian-core 1.0.10

Bug in set_cutoff - xapian-core 1.0.10

We have a bug when calling set_cutoff and
set_sort_by_value_then_relevance functions. Some documents are
displaying at the beginning and at end of result sets but are not
displaying in middle of result set.

Example: Running query with approximate result set 500,000 with
set_cutoff(97) using set_sort_by_value_then_relevance will produce
result and make document data accessible for the first 29800
documents, when going beyond asking for documents data between
29,800-30,000 these documents are not accessible creating a blank gap.
When calling data for 31,000 document, data is again available.

I would assume that if my result set is 500,000 documents and will
set_cutoff(97) I would be able to get all the documents up to certain
point, then once reach first document with no data available, I would
not expect in getting more documents after 200 documents not
accessible.

Thanks, Kevin Duraj
http://myhealthcare.com

On Tue, Jan 13, 2009 at 12:18:33PM -0800, Kevin Duraj
wrote:
> We have a bug when calling set_cutoff and
> set_sort_by_value_then_relevance functions. Some documents are
> displaying at the beginning and at end of result sets but are not
> displaying in middle of result set.
I think this is likely the same issue as this bug:

http://trac.xapian.org/ticket/216

I wrote a fix for this (aside from one corner case) last month and
attached a patch to the ticket - could you try it and see if it fixes
your problem:

http://trac.xapian.org/raw-attachment/ticket/216/bug216-fix.patch

We've been considering whether to apply this for 1.0.x - previously the
bug had only been noted from reading the code, and it means we do more
work (calculating of weights for documents we're about to discard)
adding overhead.

If it does fix your problem, I'm inclined to apply the patch for 1.0.11.

Cheers,
    Olly