Javier Garcia <javiertury at gmail.com> writes:> I've applied the path to notmuch 0.26.1 without success. > > $ rm -rf ~/.mail/.notmuch > $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH > ./notmuch new > Found 20065 total files (that's not much mail). > Processed 20065 total files in 58s (341 files/sec.). > Added 19605 new messages to the database. > > $ xapian-check .mail/.notmuch/xapian/ > docdata: > blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 > B-tree checked okay > docdata table structure checked OK > termlist: > blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 > xapian-check: DatabaseError: 1 unused block(s) missing from the free > list, first is 0OK, so probably not related to reference loops (although that patch is not very well tested). It's not clear how notmuch is triggering it, but this looks like the same bug in Xapian that olly fixed recently [1]. A possible next step is to try building xapian master, and linking notmuch against that. Maybe Patrick or Justus (in copy) has some idea why you're only seeing problems in afew. Another debugging direction is to try to duplicate your problem with some subset of mail that you're willing to share (bisection is the usual strategy). [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html
Unfortunately I can't share my emails without the approval of other parties. The minimum subsets that trigger the error are in the range of 1000-5000 mails, so asking each and everyone of them is out of my reach. I tried to replicate the problem using just spam folders without success. The following is a solid workaround I've stumbled upon. Afew no longer complains and database corruption is gone. $ notmuch compact $ xapian-check ~/.mail/.notmuch/xapian <check messages> No errors found I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it. I will wait and test again in a few weeks. If you are interested in my setup, the error happens with this minimal configuration. #~/.config/afew/config [Filter.1] query = 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)' tags = +inbox;-new message = Messages in INBOX folder are tagged as inbox [Filter.2] query = '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)' tags = -inbox message = Messages not in INBOX folder cannot be inbox #~/.notmuch-config [database] path=/home-path/.mail [new] tags=new On 07/04/18 12:51, David Bremner wrote:> Javier Garcia <javiertury at gmail.com> writes: > >> I've applied the path to notmuch 0.26.1 without success. >> >> $ rm -rf ~/.mail/.notmuch >> $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH >> ./notmuch new >> Found 20065 total files (that's not much mail). >> Processed 20065 total files in 58s (341 files/sec.). >> Added 19605 new messages to the database. >> >> $ xapian-check .mail/.notmuch/xapian/ >> docdata: >> blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 >> B-tree checked okay >> docdata table structure checked OK >> termlist: >> blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 >> xapian-check: DatabaseError: 1 unused block(s) missing from the free >> list, first is 0 > OK, so probably not related to reference loops (although that patch is > not very well tested). It's not clear how notmuch is triggering it, but > this looks like the same bug in Xapian that olly fixed recently [1]. > > A possible next step is to try building xapian master, and linking > notmuch against that. > > Maybe Patrick or Justus (in copy) has some idea why you're only seeing > problems in afew. > > Another debugging direction is to try to duplicate your problem with > some subset of mail that you're willing to share (bisection is the usual > strategy). > > [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html
Javier Garcia <javiertury at gmail.com> writes:> Unfortunately I can't share my emails without the approval of other > parties. The minimum subsets that trigger the error are in the range of > 1000-5000 mails, so asking each and everyone of them is out of my reach. > I tried to replicate the problem using just spam folders without success. > > The following is a solid workaround I've stumbled upon. Afew no longer > complains and database corruption is gone. > > $ notmuch compact > $ xapian-check ~/.mail/.notmuch/xapian > <check messages> > No errors foundRight, I should have thought of compaction, that's a workaround Olly mentioned before. That strongly suggests that you are hitting the known Xapian bug. d
Hi notmuch developers, I also had this database corruption, I waited for the fix to land in notmuch 0.26.2, build it, moved the xapian directory away, did a notmuch new and restored the tags from a dump. But the problem remains: ~$ xapian-check ~/Mail/.notmuch/xapian docdata: blocksize=8K items=10841 firstunused=75 revision=82 levels=1 root=2 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=1893162 firstunused=368983 revision=82 levels=3 root=177608 xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0 this is very similar to the old database which I had moved away: ~$ xapian-check ~/Mail/.notmuch/xapian-2018-04-29-00-22/ docdata: blocksize=8K items=10863 firstunused=78 revision=59623 levels=1 root=2 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=1894648 firstunused=360821 revision=59623 levels=3 root=360580 xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0 Now I did notmuch compact and the database check says there are no errors. This seems to me as if the fix had not helped or there is another problem. $ notmuch --version notmuch 0.26.2+26~g9e158fb ~$ xapian-compact --version xapian-compact - xapian-core 1.4.3 Thanks for developing notmuch, Gregor * Javier Garcia <javiertury at gmail.com> [2018-04-07; 17:09]:> Unfortunately I can't share my emails without the approval of other > parties. The minimum subsets that trigger the error are in the range of > 1000-5000 mails, so asking each and everyone of them is out of my reach. > I tried to replicate the problem using just spam folders without success. > > The following is a solid workaround I've stumbled upon. Afew no longer > complains and database corruption is gone. > > $ notmuch compact > $ xapian-check ~/.mail/.notmuch/xapian > <check messages> > No errors found > > I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it. > I will wait and test again in a few weeks. > > If you are interested in my setup, the error happens with this minimal > configuration. > > #~/.config/afew/config > [Filter.1] > query = 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)' > tags = +inbox;-new > message = Messages in INBOX folder are tagged as inbox > > [Filter.2] > query = '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)' > tags = -inbox > message = Messages not in INBOX folder cannot be inbox > > #~/.notmuch-config > [database] > path=/home-path/.mail > [new] > tags=new > > On 07/04/18 12:51, David Bremner wrote: >> Javier Garcia <javiertury at gmail.com> writes: >> >>> I've applied the path to notmuch 0.26.1 without success. >>> >>> $ rm -rf ~/.mail/.notmuch >>> $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH >>> ./notmuch new >>> Found 20065 total files (that's not much mail). >>> Processed 20065 total files in 58s (341 files/sec.). >>> Added 19605 new messages to the database. >>> >>> $ xapian-check .mail/.notmuch/xapian/ >>> docdata: >>> blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 >>> B-tree checked okay >>> docdata table structure checked OK >>> termlist: >>> blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 >>> xapian-check: DatabaseError: 1 unused block(s) missing from the free >>> list, first is 0 >> OK, so probably not related to reference loops (although that patch is >> not very well tested). It's not clear how notmuch is triggering it, but >> this looks like the same bug in Xapian that olly fixed recently [1]. >> >> A possible next step is to try building xapian master, and linking >> notmuch against that. >> >> Maybe Patrick or Justus (in copy) has some idea why you're only seeing >> problems in afew. >> >> Another debugging direction is to try to duplicate your problem with >> some subset of mail that you're willing to share (bisection is the usual >> strategy). >> >> [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html > > _______________________________________________ > notmuch mailing list > notmuch at notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuchCiao; Gregor -- -... --- .-. . -.. ..--.. ...-.-