similar to: xapian 1.4 performance issue

Olly Betts writes: > On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > > Recoll builds snippets by partially reconstructing documents out of index > > contents. > > > [...] > > > > The specific operation which has become slow is opening many term position > > lists, each quite short. > > The difference will actually

On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > Recoll builds snippets by partially reconstructing documents out of index > contents. > [...] > > The specific operation which has become slow is opening many term position > lists, each quite short. The difference will actually be chert vs glass, rather than 1.2 vs 1.4 as such (glass is the new backend in

Hi, I ran some tests with Recoll to compare Xapian 1.2.22 and 1.3.5 performance. I mostly used two relatively small document sets (realistic/typical recoll data subsets). The first set is a 2.2 GB mbox folder, with approximately 56K messages in 275 files, producing approximately 64K documents (because of attachments). The second set is a 11 GB folder with 5300 PDF files in it (random PDFS

Olly Betts writes: > On Mon, Apr 11, 2016 at 09:54:36AM +0200, Jean-Francois Dockes wrote: > > The question which remains for me is if I should run xapian-compact > > after an initial indexing operation. I guess that this depends on the > > amount of expected updates and that there is no easy answer ? > > I think it's not obvious whether it's a good plan

Olly Betts writes: > On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > > Some might notice the 50% index size increase. Excessive index size is > > already one relatively rare, but recurring complaint. Except if I did > > something wrong: I'm actually quite surprised by it. > > Did you try compacting the resulting databases? > >

I hadn't noticed any interactive slowdown, but when I got around to running the notmuch performance suite, there seems to be some noticable slowdown with the glass backend (default in Xapian 1.3.5) compared to chert (using xapian 1.2.22) These tests are on an older i7 with 12G of RAM and an SSD. I'm reasonable confident they are CPU bound. One curious thing is the increase in system time

On Sat, Mar 25, 2017 at 06:36:25PM -0500, Ryan Cross wrote: > After upgrades my stack is now: > > Python 2.7 > Django 1.8 > Haystack 2.6.0 > Xapian 1.4.3. (latest xapian haystack backend with some modifications) > > Using the same rebuild command as below but with —batch-size=50000 > > The issue has now become one of performance. I am indexing 2.2 million >

People, I sent this note to JF at Recoll and he suggested asking here (his response below) - any suggestions? Thanks, Phil. -------- Original Message -------- Subject: Re: Another ue for Recoll? - AI/Eliza Date: 2012-12-23 19:22 From: jf at dockes.org To: <phil at pricom.com.au> Philip Rhoades writes: > Jean, > > I have been using Recoll happily for some time now but I

Hello, Since the migration to xapian 1.4.X and the migration to Glass, i've recurrent database corruption errors. I've to reindex my database to make it works again. > DatabaseCorruptError: Expected block 200713 to be level 2, not 1 Theses problems always occur after this error: > xapian.DatabaseCorruptError: Db block overwritten - are there multiple writers? I've

Hi, I have a user reporting the following error during recoll indexing: flush() failed: Db block overwritten - are there multiple writers? "flush() failed" is from recoll, the rest is, I think the text of the Xapian exception. This is with Xapian 1.4.3 on Linux (I asked for more details, should be coming). I don't think that I've ever seen this error, and I also

On 14/09/2018 at 09:30, Jean-Francois Dockes wrote: > Hi, > > You may be interested by how Recoll does it: > > https://www.lesbonscomptes.com/recoll/idxthreads/threadingRecoll.html > > A few things in the document are slightly obsolete (esp. the last > paragraph: recollindex now does use vfork()), but it's overall quite close > to how the current indexer works.

Olly Betts <olly at survex.com> writes: > > So the T00-new.sh numbers make sense - there's more work to do, and > we need to read existing positional data more to insert the new stuff, > so the increased reads and writes make sense. > > But guessing at what the other two tests do, I wouldn't expect them to > be affected by this. The non-optimized-away cases of

On Fri, Dec 08, 2017 at 11:08:00AM +0100, Jean-Francois Dockes wrote: > This is the only really short term solution: any other is weeks or months > away. Is the "stub database" feature the appropriate way to create Chert > databases with Xapian 1.4 ? With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when constructing the WritableDatabase object. I noticed recently

On Mon, Jul 09, 2018 at 10:29:18AM +0100, Olly Betts wrote: > The attached patch reset this cursor each time commit() is called, and > that fixes my C++ reproducer, though I think this ought to work as-is > and the real bug is at a lower level. I've dug deeper and that was indeed the case. Here's a patch which addresses the root cause:

Hi Olly, Thanks for the detailed response. I hadn’t realized there was a new xapian haystack backend. I’m going to try that but I have some upgrades to do first. Django 1.8, etc. Thanks, Ryan > On Feb 28, 2017, at 3:40 PM, Olly Betts <olly at survex.com> wrote: > > On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote: >> I am trying to rebuild an index of 2+

A Recoll user gets the following message while indexing: "Attempted to delete or modify an entry in a non-existent posting list for #bannerholder" The exception happens during a commit call. Xapian version 1.4.11, Debian Buster A little more detail here: https://opensourceprojects.eu/p/recoll1/tickets/108/ I asked if this was reproducible, and to run the indexing in single-thread

Hi! Suppose you have a huge notmuch/Xapian database, built on a 32-bit system (well, actually on x86_64-pc-linux-gnu, but using a years old 32-bit notmuch binary; notmuch 0.9, Xapian 1.2.21 -- don't laugh), and suppose you're finally going to update that years old notmuch installation (release by release, forward-porting a bunch of patches). Naturally, I'd now do a native 64-bit

On Mon, Mar 05, 2018 at 09:48:52PM +0000, Olly Betts wrote: > On Mon, Mar 05, 2018 at 08:52:47PM +0100, Sylvain Taverne wrote: > > I've remarked the error occur when i'm trying to get stored values from a > > database with a lot of stored values. I can reproduce the error with simple > > python2 script i've posted on github > > > >

Dear XAPIAN developers, I was using xapian to index large than 13 million document about Q & A(Quora similarly). I will share some performance data about indexing and searching, and I will seek some help for improving performance of searching. My computer has 8 i7 at 3.4G CPU and 16G memory, ubuntu 16.04. Dataset include about 13M document, each document will be cut into 35 term(Chinese

Olly Betts writes: > On Wed, May 17, 2017 at 09:08:32PM +0200, Jean-Francois Dockes wrote: > > I have a user reporting the following error during recoll indexing: > > > > flush() failed: Db block overwritten - are there multiple writers? > > > > "flush() failed" is from recoll, the rest is, I think the text of the Xapian > > exception.