Xapian discuss - Mar 2018 - Xapian 1.4.5 "Db block overwritten - are there multiple writers?" with Glass

Hello,
Since the migration to xapian 1.4.X and the migration to Glass, i've
recurrent database corruption errors. I've to reindex my database to make
it works again.

    > DatabaseCorruptError: Expected block 200713 to be level 2, not 1

Theses problems always occur after this error:

    > xapian.DatabaseCorruptError: Db block overwritten - are there
multiple writers?

I've remarked the error occur when i'm trying to get stored values from
a
database with a lot of stored values. I can reproduce the error with simple
python2 script i've posted on github

    https://github.com/staverne/xapian_test
    https://github.com/staverne/xapian_test/blob/master/test_xapian.py

The script always works with Chert backend but sometimes fail with Glass
backend.
It look's like the error report here:

    https://lists.xapian.org/pipermail/xapian-discuss/2017-May/009525.html

Any ideas ?
Cheers,
Sylvain

On Mon, Mar 05, 2018 at 08:52:47PM +0100, Sylvain Taverne
wrote:
> I've remarked the error occur when i'm trying to get stored values
from a
> database with a lot of stored values. I can reproduce the error with simple
> python2 script i've posted on github
> 
>     https://github.com/staverne/xapian_test
>     https://github.com/staverne/xapian_test/blob/master/test_xapian.py
> 
> The script always works with Chert backend but sometimes fail with Glass
> backend.
Thanks, that's brilliant - I can reproduce with glass which means I should
be able to pin down the cause (though it's likely to take a while as it
took about 45 minutes to fail).
> It look's like the error report here:
> 
>     https://lists.xapian.org/pipermail/xapian-discuss/2017-May/009525.html
Yes, it seems likely it's the same problem.  I don't think we heard
anything beyond the end of that thread for that case though.

Cheers,
    Olly

On Mon, Mar 05, 2018 at 09:48:52PM +0000, Olly Betts
wrote:
> On Mon, Mar 05, 2018 at 08:52:47PM +0100, Sylvain Taverne wrote:
> > I've remarked the error occur when i'm trying to get stored
values from a
> > database with a lot of stored values. I can reproduce the error with
simple
> > python2 script i've posted on github
> > 
> >     https://github.com/staverne/xapian_test
> >     https://github.com/staverne/xapian_test/blob/master/test_xapian.py
> > 
> > The script always works with Chert backend but sometimes fail with
Glass
> > backend.
> 
> Thanks, that's brilliant - I can reproduce with glass which means I
should
> be able to pin down the cause (though it's likely to take a while as it
> took about 45 minutes to fail).
Just to update the status of this - I now have a C++ reproducer, and
have found that the transaction isn't needed to reproduce this.  It also
still reproduces when run under eatmydata [1], and together these bring
the time to reproduce down to 4-5 minutes.  Still longer than ideal, but
quick enough to start making some progress on narrowing down what's
happening.

Cheers,
    Olly

[1] https://www.flamingspork.com/projects/libeatmydata/