Xapian discuss - Mar 2010 - "Value in posting list too large" error with 1.1.4 (chert and brass, not flint)

Hi,

I've a program which:

     1. Sets XAPIAN_FLUSH_THRESHOLD=1000
     2. Opens a (new) database for write
     3. Indexes a few thousand documents
     4. Periodically also does queries on the database

With 1.1.4, with certain document sets (basically a particular mail
folder of mine), Enquire.get_mset() sometimes (but not always) triggers
a "RangeError: Value in posting list too large" error.  Once this
happens, occasionally I also see DocNotFoundError.

After the indexing completes and a final explicit flush() is called and
the program exits, xapian-check reports no errors found, so at least the
database seems consistent on disk.

What's interesting is that if I set XAPIAN_FLUSH_THRESHOLD to be a value
larger than the document set size (and I am not explicitly flushing
either), the problem does not occur.

The problem occurs with the chert and brass backends, but not flint.
(Nor does it, perhaps obviously, occur with xapian-core 1.0.16.)  Just
to rule it out, I also tried 1.1.4 built with gcc 4.1 (rather than 4.2
with -fno-strict-aliasing) but the result was the same.

I am trying to reproduce the problem as a smaller test case, but haven't
been terribly successful so far.  I was hoping that someone might have
some ideas about possible explanations that might help me work up a
useful test case.

Thanks,
Jason.

On Sun, 2010-03-07 at 15:58 -0500, Jason Tackaberry
wrote:
> I am trying to reproduce the problem as a smaller test case, but
haven't
> been terribly successful so far.  I was hoping that someone might have
> some ideas about possible explanations that might help me work up a
> useful test case.
A bit more digging reveals that if I strip out all non-word characters
from the query I was periodically performing, the problem goes away.

I _was_ using the query parser.  I'm now constructing the query directly
(it's a phrase search), which is obviously better anyway.

But it appears there might be something about non-word characters that
isn't being handled robustly in chert and brass.

Frustratingly, even with that information I can't seem to generate a
simple test case to trigger the problem.

On Sun, Mar 07, 2010 at 06:08:36PM -0500, Jason Tackaberry
wrote:
> On Sun, 2010-03-07 at 22:57 +0000, Olly Betts wrote:
> > It seems to me this is certain to be a red-herring.  Except for zero
bytes
> > the characters in a term just a string of bytes to the backend.
> > 
> > So it's just coincidence that changing the query avoids the
problem.
> 
> Can you elaborate on what the error means and what the likely causes
> are?
Yes, but that requires me to look at the code.  I just addressed this point
directly first to avoid you wasting time on it.

It means we tried to unpack a variable length integer, but the encoded value
was larger than the type can hold.  That probably means we're trying to
decode
the wrong data, or junk.  This exception could come from many places though.

I assume since the test data is email that it's not something you can make
available?

If not, you'll need to do the debugging leg work.  I'd suggest trying
trunk
first.  I don't see a relevant fix, but it would be annoying to spend time
debugging an already fixed problem.  You can use a snapshot tarball if you
don't fancy bootstrapping from SVN - see near the bottom of:

http://xapian.org/bleeding

If you configure xapian-core without optimisation and with debug info for
easier debugging:

./configure CXXFLAGS='-O0 -g'

Build it, and then run your test program under gdb, e.g. for omindex:

gdb --args omindex --db /path/to/db /path/to/html

Then set a breakpoint on where this exception is thrown, and run it:

b backends/chert/chert_postlist.cc:153
(y to confirm)
run

When gdb stops at that line, get a backtrace with "bt".

Cheers,
    Olly