similar to: Optimal usage of xapian-compact for merging

Greets, I've encountered the following while performing test merges (and writing code to handle errors, etc so things can be automated) and wondering about the best way to proceed: xapian-compact -b64k -m src1 src2.... tmp_dst -- works as expected, exit code 0. xapian-check tmp_dst -- produces the following error for the postlist: postlist: baseB blocksize=64K items=28175410

On Tue, March 23, 2010 19:46, Kevin Duraj wrote: > I am merging 300 indexes at once, it takes less than a day for merge > to happen for 100 million documents, during merging I notice very heavy IO. That IO sounds pretty normal. To help with IO load, we have a dedicated index store cluster, dedicated source data cluster, dedicated indexing cluster, etc. Sigh. Each time I think we have

Greetings, Using latest svn. I've noticed the following error when performing index merging: postlist: baseB blocksize=8K items=33962 lastblock=534 revision=1 levels=2 root=459 B-tree checked okay Tag containing meta information is corrupt. postlist table errors found: 1 I can still search on this index (I've only checked very small indexes), but merging is now a problem since I check

Hi, I have a user reporting the following error during recoll indexing: flush() failed: Db block overwritten - are there multiple writers? "flush() failed" is from recoll, the rest is, I think the text of the Xapian exception. This is with Xapian 1.4.3 on Linux (I asked for more details, should be coming). I don't think that I've ever seen this error, and I also

Running notmuch from git on Debian testing[1] with the mail and database sitting on a ZFS filesystem, adding mail to a new database: > agrajag-testing ~/s/notmuch % ./notmuch new > Found 605510 total files (that's not much mail). > add_file: A Xapian exception occurred36m 37s remaining). > A Xapian exception occurred adding message: Unexpected end of posting list for

A Recoll user gets the following message while indexing: "Attempted to delete or modify an entry in a non-existent posting list for #bannerholder" The exception happens during a commit call. Xapian version 1.4.11, Debian Buster A little more detail here: https://opensourceprojects.eu/p/recoll1/tickets/108/ I asked if this was reproducible, and to run the indexing in single-thread

Mike Hommey <mh at glandium.org> writes: > Hi, > > When running `notmuch compact` today, it stopped with the following > output: > > Compacting database... > compacting table postlist > Reduced by 25% 648656K (2498904K -> 1850248K) > compacting table docdata > Reduced by 15% 24K (152K -> 128K) > compacting table termlist > Reduced by

Olly Betts writes: > On Wed, May 17, 2017 at 09:08:32PM +0200, Jean-Francois Dockes wrote: > > I have a user reporting the following error during recoll indexing: > > > > flush() failed: Db block overwritten - are there multiple writers? > > > > "flush() failed" is from recoll, the rest is, I think the text of the Xapian > > exception.

what is the fastest way to fetch results which are sorted by timestamp ? i want to use xapian as my search engine , use add_boolean_term(something) and add_value(0,sortable_serialise(get_timestamp())) to a doc. search through enquire.set_weighting_scheme(xapian.BoolWeight()) and enquire.set_sort_by_value(0,True) to ensure that the results are sorted by the timestamp. This method is ok , but

Hi, We were running xapian-check on one of our Xapian indexes and it returns the following error: position: baseB blocksize=8K items=809896869 lastblock=2090419 revision=3161 levels=3 root=2084903 Failed to check B-tree: DatabaseError: Items not in sorted order The other tables verify without issue. It looks like our oldest backup of this database (a month old) has the same issue. Searching and

Greets, When merging lots of subindexes in batches like so: xapian-compact -m idx1 idx2... dstidx Errors such as: xapian-compact: DatabaseError: Error reading block 0: got end of file present a problem since it does not provide the offending path name (of the broken index) for easy identification/removal in automated/batch scenarios (the way DatabaseOpeningError:.... does, eg). The only way

Greets, I've had xapian-compact (without -F) sessions running for several days now on 10 'merge' machines and I've noticed that the average compaction average can swing wildly: 18% 76% 10% 19% 39% 13% 69% 43% 19% 42% The average so far is about 35% (ie, 65% reduction in target index sizes, which is unexpected and pleasingly welcomed). I'm curious about the large variance in

I am fairly confident that these issues are related to killing the scriptindex process ungracefully causing blocks that were queued for writing to disk to not get written. I mention to send you the file because it could be that you would see almost immediately with the situation is. Thanks > ----- Original Message ----- > From: oscaruser@programmer.net > To:

Greetings, I've been wondering about the index format and backwards compatibility. We're using the dev version (for chert) and each svn up means that any indexes created prior to this revision cannot be read. Is this purely a cautious move to prevent errors, and, barring any obvious index format changes, can I safely force the current revision to read existing indexes? eg, by

Dear XAPIAN developers, I was using xapian to index large than 13 million document about Q & A(Quora similarly). I will share some performance data about indexing and searching, and I will seek some help for improving performance of searching. My computer has 8 i7 at 3.4G CPU and 16G memory, ubuntu 16.04. Dataset include about 13M document, each document will be cut into 35 term(Chinese

On Mon, Apr 09, 2007 at 11:26:44AM +0100, richard wrote: > backends/flint/flint_database.cc: Fix bug with document lengths > in replace document. Was using doclens.insert() to set the new > document length, but this has no effect if the entry already > exists. Argh! That issue has bitten us at least once before. I'll audit all uses of insert to see if there are any other

Matt <mattator at gmail.com> writes: > thanks didn't know about xapian-check ! > the output > === > docdata: > blocksize=8K items=70 firstunused=3 revision=421 levels=0 root=2 > B-tree checked okay > docdata table structure checked OK > > termlist: > blocksize=8K items=186136 firstunused=62058 revision=421 levels=2 root=12260 > B-tree checked okay >

On Wed, Jun 19, 2013, at 03:49 PM, Olly Betts wrote: > On Wed, Jun 19, 2013 at 01:29:16PM +1000, Bron Gondwana wrote: > > The advantage of compact - it runs approximately 8 times as fast (we > > are CPU limited in each case - writing to tmpfs first, then rsyncing > > to the destination) and it takes approximately 75% of the space of a > > fresh database with maximum

On Fri, Jul 29, 2005 at 10:08:13AM +0100, james wrote: > SVN root: svn://svn.xapian.org/xapian > Changes by: james > Revision: 6355 > Date: 2005-07-29 10:08:13 +0100 (Fri, 29 Jul 2005) > > Log message (6 lines): > omindex.cc: add --preserve-nonduplicates / -p option to not delete any > documents that aren't updated, in replace duplicates mode

I'm trying to compact (or at least merge) multiple databases, while stripping search records which are no longer required. Backstory: I've inherited the Cyrus IMAPd xapian-based search code from Greg Banks when he left Opera. One of the unfinished parts was removing expunged emails from the search database. We moved from having a single search database to supporting multiple