similar to: Xapian 1.3.5 snapshot performance and index size

Olly Betts writes: > On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > > Some might notice the 50% index size increase. Excessive index size is > > already one relatively rare, but recurring complaint. Except if I did > > something wrong: I'm actually quite surprised by it. > > Did you try compacting the resulting databases? > >

Hi, I have had reports that Recoll has become unbearingly slow in some instances. After inquiry, this happens with Xapian 1.4 only, and the part which does not work any more is the snippets extraction. Recoll builds snippets by partially reconstructing documents out of index contents. For this, after determining a set of document term positions to be displayed (around the hopefully interesting

I hadn't noticed any interactive slowdown, but when I got around to running the notmuch performance suite, there seems to be some noticable slowdown with the glass backend (default in Xapian 1.3.5) compared to chert (using xapian 1.2.22) These tests are on an older i7 with 12G of RAM and an SSD. I'm reasonable confident they are CPU bound. One curious thing is the increase in system time

On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > Some might notice the 50% index size increase. Excessive index size is > already one relatively rare, but recurring complaint. Except if I did > something wrong: I'm actually quite surprised by it. Did you try compacting the resulting databases? Creating a database by calling add_document() repeatedly would have

On Sat, Mar 25, 2017 at 06:36:25PM -0500, Ryan Cross wrote: > After upgrades my stack is now: > > Python 2.7 > Django 1.8 > Haystack 2.6.0 > Xapian 1.4.3. (latest xapian haystack backend with some modifications) > > Using the same rebuild command as below but with —batch-size=50000 > > The issue has now become one of performance. I am indexing 2.2 million >

Olly Betts writes: > On Mon, Apr 11, 2016 at 09:54:36AM +0200, Jean-Francois Dockes wrote: > > The question which remains for me is if I should run xapian-compact > > after an initial indexing operation. I guess that this depends on the > > amount of expected updates and that there is no easy answer ? > > I think it's not obvious whether it's a good plan

On Thu, Apr 07, 2016 at 08:56:46AM -0300, David Bremner wrote: > I hadn't noticed any interactive slowdown, but when I got around to > running the notmuch performance suite, there seems to be some noticable > slowdown with the glass backend (default in Xapian 1.3.5) compared to > chert (using xapian 1.2.22) Some of this is pretty much expected, though other parts I don't

Hi, We were running xapian-check on one of our Xapian indexes and it returns the following error: position: baseB blocksize=8K items=809896869 lastblock=2090419 revision=3161 levels=3 root=2084903 Failed to check B-tree: DatabaseError: Items not in sorted order The other tables verify without issue. It looks like our oldest backup of this database (a month old) has the same issue. Searching and

Olly Betts writes: > On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > > Recoll builds snippets by partially reconstructing documents out of index > > contents. > > > [...] > > > > The specific operation which has become slow is opening many term position > > lists, each quite short. > > The difference will actually

On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > Recoll builds snippets by partially reconstructing documents out of index > contents. > [...] > > The specific operation which has become slow is opening many term position > lists, each quite short. The difference will actually be chert vs glass, rather than 1.2 vs 1.4 as such (glass is the new backend in

On 21/09/2018 at 08:03, Olly Betts wrote: > > You can then either just search those databases together or use > Xapian::Database::compact() (or the xapian-compact command line tool) to > merge the databases once built. > just my 2 cents tips: a commit() to the database is needed before compact() otherwise every attempt to search fails. This on GNU Linux Debian 9.5 Xapian:

Dear senior ML members and developers of Xapian Omega, Mr. Olly has helped me cross the bump of the initial learning curve. (ref: https://lists.xapian.org/pipermail/xapian-discuss/2024-April/010034.html) How can I use Xapian Omega directly (i.e., without using `recoll` and `xapiandb`) to index a directory of text files with all strings greater than 3 characters, to create an index text file

Hello friends, I have installed xapian-omega for my system installed from Official Debian GNU/Linux Live 11.6.0 lxde 2022-12-17T11:46, It is an old edition but I am okay with it, I tasted the power of xapian while using recoll. I read that xapian is a powerful database building package for character strings, and indexes a directory of files really well. I want to use xapian-omega as a customisable

Hi Olly, Thanks for the detailed response. I hadn’t realized there was a new xapian haystack backend. I’m going to try that but I have some upgrades to do first. Django 1.8, etc. Thanks, Ryan > On Feb 28, 2017, at 3:40 PM, Olly Betts <olly at survex.com> wrote: > > On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote: >> I am trying to rebuild an index of 2+

I have installed the new Xapian 1.4.0 , during the installation, I haven't seen any problems, however, when I execute commands quest and delve I get different versions, and my Perl-based searches return Exception: Couldn't detect type of database ... and what are these glass things in the index directories? There is a no new version of Perl Search::Xapian. $ quest -version quest -

Hello, Since the migration to xapian 1.4.X and the migration to Glass, i've recurrent database corruption errors. I've to reindex my database to make it works again. > DatabaseCorruptError: Expected block 200713 to be level 2, not 1 Theses problems always occur after this error: > xapian.DatabaseCorruptError: Db block overwritten - are there multiple writers? I've

Dear XAPIAN developers, I was using xapian to index large than 13 million document about Q & A(Quora similarly). I will share some performance data about indexing and searching, and I will seek some help for improving performance of searching. My computer has 8 i7 at 3.4G CPU and 16G memory, ubuntu 16.04. Dataset include about 13M document, each document will be cut into 35 term(Chinese

I came across this benchmark between Xapian & Solr: http://www.anur.ag/blog/2009/03/xapian-and-solr/ According to the benchmark, a doc set that took Solr 34 min to index took Xapian 7 hours. Solr's index is also much smaller - 2.5GB to Xapian's 8.9GB. I'm new to Xapian. Just wondering if results like these are typical? Is indexing speed & size a known issue in Xapian? Or is

Hi! Suppose you have a huge notmuch/Xapian database, built on a 32-bit system (well, actually on x86_64-pc-linux-gnu, but using a years old 32-bit notmuch binary; notmuch 0.9, Xapian 1.2.21 -- don't laugh), and suppose you're finally going to update that years old notmuch installation (release by release, forward-porting a bunch of patches). Naturally, I'd now do a native 64-bit

Olly Betts writes: > On Mon, Jan 21, 2019 at 03:25:01PM +0100, Jean-Francois Dockes wrote: > > I have had a problem report from a Recoll user about the amount of writes > > during index creation. > > > > https://opensourceprojects.eu/p/recoll1/tickets/67/ > > > > The issue is that the index is on SSD and that the amount of writes is > >