similar to: Segmentation fault in matcher/queryoptimiser

Olly, thanks for your feedback. On Mon, Jul 31, 2017, at 23:29, Olly Betts wrote: > On Mon, Jul 31, 2017 at 09:24:29AM +0200, Robert Stepanek wrote: > > We'd appreciate any hints on how to fix this. I've written up our > > findings and solution attempts below. Should we post this on trac? > > Yes, it'd be good to have a ticket to track this. I've created

Dear All, I've tried to build xapian-core 1.4.3 on MSYS2. It fails with attached error (undefined reference to `exp10'). I think it might be missing an explicit link to 'm'. I'm not able to fix this myself as I do not know autotools sufficiently well, but I hope you might be able to help. Cheers, Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel.

Hello, There is a memory leak in Xapian 1.2.4. We use a persistant connection in FastCGI processes. As soon as we catch this exception, "dmalloc" recognizes memory leaks: The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation Down below the output of "dmalloc". This happens only on the production system. On my

Olly Betts <olly at survex.com> wrote: > On Fri, Aug 21, 2020 at 09:06:59AM +0000, Eric Wong wrote: > > Going back to the "prioritizing aggregated DBs" thread from > > February 2020, I've got 390 Xapian shards for 130 public inboxes > > I want to search against(*). There's more on the horizon (we're > > expecting tens of thousands of public

Olly Betts writes: > On Mon, Apr 11, 2016 at 09:54:36AM +0200, Jean-Francois Dockes wrote: > > The question which remains for me is if I should run xapian-compact > > after an initial indexing operation. I guess that this depends on the > > amount of expected updates and that there is no easy answer ? > > I think it's not obvious whether it's a good plan

On Wed, Jun 19, 2013, at 03:49 PM, Olly Betts wrote: > On Wed, Jun 19, 2013 at 01:29:16PM +1000, Bron Gondwana wrote: > > The advantage of compact - it runs approximately 8 times as fast (we > > are CPU limited in each case - writing to tmpfs first, then rsyncing > > to the destination) and it takes approximately 75% of the space of a > > fresh database with maximum

Going back to the "prioritizing aggregated DBs" thread from February 2020, I've got 390 Xapian shards for 130 public inboxes I want to search against(*). There's more on the horizon (we're expecting tens of thousands of public inboxes). After bumping RLIMIT_NOFILE and running ->add_database a bunch, the actual queries seem to be taking ~30s (not good :x). Now I'm

Greetings, Using latest svn. I've noticed the following error when performing index merging: postlist: baseB blocksize=8K items=33962 lastblock=534 revision=1 levels=2 root=459 B-tree checked okay Tag containing meta information is corrupt. postlist table errors found: 1 I can still search on this index (I've only checked very small indexes), but merging is now a problem since I check

Running notmuch from git on Debian testing[1] with the mail and database sitting on a ZFS filesystem, adding mail to a new database: > agrajag-testing ~/s/notmuch % ./notmuch new > Found 605510 total files (that's not much mail). > add_file: A Xapian exception occurred36m 37s remaining). > A Xapian exception occurred adding message: Unexpected end of posting list for

Greets, I've encountered the following while performing test merges (and writing code to handle errors, etc so things can be automated) and wondering about the best way to proceed: xapian-compact -b64k -m src1 src2.... tmp_dst -- works as expected, exit code 0. xapian-check tmp_dst -- produces the following error for the postlist: postlist: baseB blocksize=64K items=28175410

I'm trying to compact (or at least merge) multiple databases, while stripping search records which are no longer required. Backstory: I've inherited the Cyrus IMAPd xapian-based search code from Greg Banks when he left Opera. One of the unfinished parts was removing expunged emails from the search database. We moved from having a single search database to supporting multiple

I'm playing around with a machine that has 2 GB of memory. Indexing about 5GB of data average of 2MB per document. The documents are plain text. I notice the omindex's memory fott print get's biger an bigger then the machine starts to swap and it all slows down to a crawl. In regards to export XAPIAN_FLUSH_THRESHOLD I know the default is 10000 Am I right in saying that for my setup

Xapians, I am maintaining about two indexes for my search engines which approximately is each the same size. I would like to share this knowledge with you, since many of you have never seen Xapian index of this size. And of course you can search the index by yourself at - http://myhealthcare.com/ - http://find1friend.com/ I need 2 x 100 million more documents into each index, and I hope it will

olly wrote: > Log message (7 lines): > backends/database.cc: Database::Internal can't call the > PostingIterator(PostingIterator::Internal*) ctor (at least under > g++ 3.3.5) because it isn't a friend (only class Database is). For the record, Mark just reported this to me under windows so it was a problem there too, but it does work under GCC 4.1. No idea which compiler is

Hi, Following the Recoll snippets generation performance problem caused by the new positions list storage scheme in Xapian 1.4, I am experimenting with generating snippets from the complete document text stored in the index. This increases the index size much less than I would have expected (around 10-15% apparently with my home directory data), which is good news obviously. I have tried

Hello, I noticed a problem with DatabaseCorruptError exceptions with public-inbox and I guess it's user error... The problem is public-inbox was calling replace_document to modify the DB while iterating through a PostingIterator. At first I thought it was a glass problem, but I've hit it with chert on my dataset, too. I have a standalone Perl script to reproduce the problem at

Olly Betts writes: > On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > > Some might notice the 50% index size increase. Excessive index size is > > already one relatively rare, but recurring complaint. Except if I did > > something wrong: I'm actually quite surprised by it. > > Did you try compacting the resulting databases? > >

Postlists are split up into chunks, so that skip_to can avoid reading all the postlist. Currently the chunk threshold is 2048, but this is checked before adding an entry, so the postlist chunk can actually grow a little larger. Something like 2060 at most. Unfortunately this isn't a good threshold with the default blocksize (8192 bytes). Internally the B-tree splits up items with a large

Hi, The size of the string generated by VSEncoder is 12592387, while that by InterpolativeEncoder is 8554817. When only encoding the first 1000lines, both cost 0ms to decode and VS cost 1ms to encode while Interpolative cost 0ms, 1000lines is just too little to catch the difference in my test. I upload the source code to https://github.com/HurricaneTong/Xapian/tree/master/VSEncoder The

A Recoll user gets the following message while indexing: "Attempted to delete or modify an entry in a non-existent posting list for #bannerholder" The exception happens during a commit call. Xapian version 1.4.11, Debian Buster A little more detail here: https://opensourceprojects.eu/p/recoll1/tickets/108/ I asked if this was reproducible, and to run the indexing in single-thread