Hello,
I am trying to rebuild an index of 2+ million documents and have not been
successful. I am running
Python 2.7
Django 1.7
Haystack 2.1.1
Xapian 1.2.21
The index rebuild command I’m using is: django-admin.py rebuild_index --noinput
--batch-size=100000
The rebuild completes but an immediate xapian-check returns this error:
xapian-check ./archive_index
record:
baseB blocksize=8K items=2219301 lastblock=126957 revision=223 levels=2
root=126448
B-tree checked okay
record table structure checked OK
termlist:
baseB blocksize=8K items=4438602 lastblock=1134080 revision=223 levels=3
root=1128535
B-tree checked okay
termlist table structure checked OK
postlist:
baseB blocksize=8K items=34620971 lastblock=2286112 revision=223 levels=3
root=3577
B-tree checked okay
postlist table structure checked OK
position:
baseB blocksize=8K items=942481128 lastblock=2787202 revision=223 levels=3
root=2776373
xapian-check: DatabaseCorruptError: Expected block 2763648 to be level 0, not 1
Trying the latest stable version, Xapian 1.4.3, it fails during the rebuild:
All documents removed.
Indexing 2233651 messages
Traceback (most recent call last):
…
File
"/a/mailarch/current/haystack/management/commands/update_index.py",
line 221, in handle_label
self.update_backend(label, using)
File
"/a/mailarch/current/haystack/management/commands/update_index.py",
line 266, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File
"/a/mailarch/current/haystack/management/commands/update_index.py",
line 89, in do_update
backend.update(index, current_qs)
File "/a/mailarch/current/haystack/backends/xapian_backend.py", line
286, in update
database.close()
xapian.DatabaseCorruptError: Expected block 615203 to be level 0, not 1
docdata:
blocksize=8K items=380000 firstunused=21983 revision=38 levels=2 root=21410
Any suggestions for how I could get more information to troubleshoot this
failure would be
greatly appreciated.
Thank You,
Ryan
On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote:> I am trying to rebuild an index of 2+ million documents and have not been successful. I am running > > Python 2.7 > Django 1.7 > Haystack 2.1.1 > Xapian 1.2.21 > > The index rebuild command I’m using is: django-admin.py rebuild_index --noinput --batch-size=100000 > The rebuild completes but an immediate xapian-check returns this error:[...]> Trying the latest stable version, Xapian 1.4.3, it fails during the rebuild: > > All documents removed. > Indexing 2233651 messages > Traceback (most recent call last): > … > > File "/a/mailarch/current/haystack/management/commands/update_index.py", line 221, in handle_label > self.update_backend(label, using) > File "/a/mailarch/current/haystack/management/commands/update_index.py", line 266, in update_backend > do_update(backend, index, qs, start, end, total, self.verbosity) > File "/a/mailarch/current/haystack/management/commands/update_index.py", line 89, in do_update > backend.update(index, current_qs) > File "/a/mailarch/current/haystack/backends/xapian_backend.py", line 286, in update > database.close()What's the version of xapian-haystack? There's not a database.close() anywhere near line 286 in git master: https://github.com/notanumber/xapian-haystack/blob/master/xapian_backend.py#L286> xapian.DatabaseCorruptError: Expected block 615203 to be level 0, not 1 > docdata: > blocksize=8K items=380000 firstunused=21983 revision=38 levels=2 root=21410Is that the full output of xapian-check?> Any suggestions for how I could get more information to troubleshoot this > failure would be greatly appreciated.Is the data to reproduce this something you can make available? I'd stick with Xapian 1.4.3 for trying to narrow this down (if it's a Xapian bug we can backport the fix once identified). The error message means that a block which was expected to be at the leaf level was actually marked as being one level above, which suggests either there's an obscure bug in the backend code which only manifests in rare circumstances, or something is corrupting data (could be in memory or on disk). Since this happens with both 1.2.x and 1.4.x I would tend to suspect it's something external (rather than a bug in Xapian) as the default backends in 1.2 and 1.4 have some significant differences. It's certainly possible it's a Xapian bug, but if so I would expect we'd be seeing other reports, though maybe we've actually had one or two and thought them due to #675, which was fixed in 1.2.21 (however nobody's yet said "no, still seeing that"): https://trac.xapian.org/ticket/675 You could look at block 615203 of docdata.glass to see what it looks like - that might offer clues: xxd -g1 -seek $((615203*8192)) -len 8192 docdata.glass It'd also be good to eliminate possible system issues - e.g. check the disk is healthy (check the SMART status, run fsck on it), run a RAM test (distros often provide a way to run memtest86+ or similar from the boot menu). Cheers, Olly
Hi Olly, Thanks for the detailed response. I hadn’t realized there was a new xapian haystack backend. I’m going to try that but I have some upgrades to do first. Django 1.8, etc. Thanks, Ryan> On Feb 28, 2017, at 3:40 PM, Olly Betts <olly at survex.com> wrote: > > On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote: >> I am trying to rebuild an index of 2+ million documents and have not been successful. I am running >> >> Python 2.7 >> Django 1.7 >> Haystack 2.1.1 >> Xapian 1.2.21 >> >> The index rebuild command I’m using is: django-admin.py rebuild_index --noinput --batch-size=100000 >> The rebuild completes but an immediate xapian-check returns this error: > [...] >> Trying the latest stable version, Xapian 1.4.3, it fails during the rebuild: >> >> All documents removed. >> Indexing 2233651 messages >> Traceback (most recent call last): >> … >> >> File "/a/mailarch/current/haystack/management/commands/update_index.py", line 221, in handle_label >> self.update_backend(label, using) >> File "/a/mailarch/current/haystack/management/commands/update_index.py", line 266, in update_backend >> do_update(backend, index, qs, start, end, total, self.verbosity) >> File "/a/mailarch/current/haystack/management/commands/update_index.py", line 89, in do_update >> backend.update(index, current_qs) >> File "/a/mailarch/current/haystack/backends/xapian_backend.py", line 286, in update >> database.close() > > What's the version of xapian-haystack? There's not a database.close() anywhere > near line 286 in git master: > > https://github.com/notanumber/xapian-haystack/blob/master/xapian_backend.py#L286 > >> xapian.DatabaseCorruptError: Expected block 615203 to be level 0, not 1 >> docdata: >> blocksize=8K items=380000 firstunused=21983 revision=38 levels=2 root=21410 > > Is that the full output of xapian-check? > >> Any suggestions for how I could get more information to troubleshoot this >> failure would be greatly appreciated. > > Is the data to reproduce this something you can make available? > > I'd stick with Xapian 1.4.3 for trying to narrow this down (if it's a Xapian > bug we can backport the fix once identified). > > The error message means that a block which was expected to be at the leaf level > was actually marked as being one level above, which suggests either there's an > obscure bug in the backend code which only manifests in rare circumstances, or > something is corrupting data (could be in memory or on disk). > > Since this happens with both 1.2.x and 1.4.x I would tend to suspect it's > something external (rather than a bug in Xapian) as the default backends in 1.2 > and 1.4 have some significant differences. It's certainly possible it's a > Xapian bug, but if so I would expect we'd be seeing other reports, though maybe > we've actually had one or two and thought them due to #675, which was fixed in > 1.2.21 (however nobody's yet said "no, still seeing that"): > > https://trac.xapian.org/ticket/675 > > You could look at block 615203 of docdata.glass to see what it looks like - > that might offer clues: > > xxd -g1 -seek $((615203*8192)) -len 8192 docdata.glass > > It'd also be good to eliminate possible system issues - e.g. check the disk is > healthy (check the SMART status, run fsck on it), run a RAM test (distros often > provide a way to run memtest86+ or similar from the boot menu). > > Cheers, > Olly