Xapian discuss - Nov 2021 - A corrupt index

Hi,


We have an index which is around 1.1 TB on disk, which has started
giving an error when I try to replace or delete a single, specific
document:

    [M] DELETE DOCUMENT FROM INDEX 10491876
    Operation "eq": no method found,
            left argument in overloaded package Search::Xapian::DatabaseError,
            right argument has no overloaded magic at
/usr/share/perl5/MCE/Signal.pm line 306, <__ANONIO__> line 1290.

Running xapian-check gives:

    $ xapian-check ./index
    record:
    baseA blocksize=8K items=10442915 lastblock=221214 revision=5434326 levels=3
root=1332
    B-tree checked okay
    record table structure checked OK

    termlist:
    baseA blocksize=8K items=20885830 lastblock=10207839 revision=5434326
levels=3 root=7090
    Failed to check B-tree: DatabaseError: Key >= right dividing key in level
above
    postlist:
    baseA blocksize=8K items=827807972 lastblock=17007553 revision=5434326
levels=4 root=755
    B-tree checked okay
    postlist table structure checked OK

    position:
    baseA blocksize=8K items=13525517523 lastblock=118173891 revision=5434326
levels=4 root=6551
    Failed to check B-tree: DatabaseError: Table entry count says 13525517523
but actually counted 13525517278
    spelling:
    Lazily created, and not yet used.

    synonym:
    Lazily created, and not yet used.

    Total errors found: 2

Is there any chance to fix this without starting over?

If I can just delete some thousands of documents and redo those, it
would be much better for me than having to recreate the entire index.

(We have had some problems recently with the introduction of a new kind
of error, making the indexing program crash in "interesting" ways, so
I
am not in doubt that this is a "self-inflicted wound").

I think the index is chert (there is an 'iamchert' file), it's on an
Ubuntu 18.04 server with libxapian30 1.4.5-1ubuntu0.1.


  Best regards,

    Adam

-- 
 "It's the Disneyland-law                                   Adam
Sj?gren
  So far out it makes anarchy a bore"                  asjo at koldfront.dk

On Mon, Nov 15, 2021 at 09:45:15PM +0100, Adam Sj?gren
wrote:
>     termlist:
>     baseA blocksize=8K items=20885830 lastblock=10207839 revision=5434326
levels=3 root=7090
>     Failed to check B-tree: DatabaseError: Key >= right dividing key in
level above
This error is essentially saying that check found a branch node where
the key which is meant to partition two nodes below doesn't actually do
so.  Maybe replacing the bad dividing key with one which actually
partitions the nodes below (assuming they are actually partitioned)
would given a sensible working database, but it's hard to know without
trying it.
>     position:
>     baseA blocksize=8K items=13525517523 lastblock=118173891
revision=5434326 levels=4 root=6551
>     Failed to check B-tree: DatabaseError: Table entry count says
13525517523 but actually counted 13525517278
This error is just that the record of how many entries there are is
wrong (this count is stored since it's useful to know in some cases, and
expensive to compute by scanning the whole table).  It shouldn't get out
of step with the actual number of entries in the table, but since no
other errors are reported just fixing the metadata record seems
reasonable.
> Is there any chance to fix this without starting over?
Unfortunately there isn't an existing tool to try to fix these sort of
things (xapian-check has an "F" mode which can fix some chert
problems,
but they're all to do with recreating missing base and iamchert files
which is a problem sometimes seen with chert databases if a machine
loses power or hangs).

We recommend migrating off chert so perhaps reindexing is a good plan
anyway.  I can see it may not be very appealing with a 1.1TB database
though.
> (We have had some problems recently with the introduction of a new kind
> of error, making the indexing program crash in "interesting"
ways, so I
> am not in doubt that this is a "self-inflicted wound").
>
> I think the index is chert (there is an 'iamchert' file), it's
on an
> Ubuntu 18.04 server with libxapian30 1.4.5-1ubuntu0.1.
It shouldn't really be possible for the program to cause a corrupt
database like this (except for program bugs like stray memory writes
into memory Xapian has allocated, or the program writing to file
descriptors which Xapian has open for writing on the database, etc).

However, the way a chert commit happens involves trying to stitch
together per-table atomic commits to make a per-database atomic commit,
which means we need to recover if some tables have committed and others
haven't - that's fiddly to do and we've found bugs there before.  It
could be you've hit another one maybe.

In glass we replaced this whole mechanism with a new one which gives a
per-database atomic commit directly.

Cheers,
    Olly