similar to: Xapian 1.3.5 snapshot performance and index size

Displaying 20 results from an estimated 20000 matches similar to: "Xapian 1.3.5 snapshot performance and index size"

2016 Apr 11
2
Xapian 1.3.5 snapshot performance and index size
Olly Betts writes: > On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > > Some might notice the 50% index size increase. Excessive index size is > > already one relatively rare, but recurring complaint. Except if I did > > something wrong: I'm actually quite surprised by it. > > Did you try compacting the resulting databases? > >
2017 Dec 07
2
xapian 1.4 performance issue
Hi, I have had reports that Recoll has become unbearingly slow in some instances. After inquiry, this happens with Xapian 1.4 only, and the part which does not work any more is the snippets extraction. Recoll builds snippets by partially reconstructing documents out of index contents. For this, after determining a set of document term positions to be displayed (around the hopefully interesting
2016 Apr 07
2
slowdown in notmuch perf suite with xapian 1.3.5
I hadn't noticed any interactive slowdown, but when I got around to running the notmuch performance suite, there seems to be some noticable slowdown with the glass backend (default in Xapian 1.3.5) compared to chert (using xapian 1.2.22) These tests are on an older i7 with 12G of RAM and an SSD. I'm reasonable confident they are CPU bound. One curious thing is the increase in system time
2016 Apr 11
0
Xapian 1.3.5 snapshot performance and index size
On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > Some might notice the 50% index size increase. Excessive index size is > already one relatively rare, but recurring complaint. Except if I did > something wrong: I'm actually quite surprised by it. Did you try compacting the resulting databases? Creating a database by calling add_document() repeatedly would have
2017 Apr 03
3
errors on rebuild
On Sat, Mar 25, 2017 at 06:36:25PM -0500, Ryan Cross wrote: > After upgrades my stack is now: > > Python 2.7 > Django 1.8 > Haystack 2.6.0 > Xapian 1.4.3. (latest xapian haystack backend with some modifications) > > Using the same rebuild command as below but with —batch-size=50000 > > The issue has now become one of performance. I am indexing 2.2 million >
2016 Apr 12
2
Xapian 1.3.5 snapshot performance and index size
Olly Betts writes: > On Mon, Apr 11, 2016 at 09:54:36AM +0200, Jean-Francois Dockes wrote: > > The question which remains for me is if I should run xapian-compact > > after an initial indexing operation. I guess that this depends on the > > amount of expected updates and that there is no easy answer ? > > I think it's not obvious whether it's a good plan
2016 Apr 07
0
slowdown in notmuch perf suite with xapian 1.3.5
On Thu, Apr 07, 2016 at 08:56:46AM -0300, David Bremner wrote: > I hadn't noticed any interactive slowdown, but when I got around to > running the notmuch performance suite, there seems to be some noticable > slowdown with the glass backend (default in Xapian 1.3.5) compared to > chert (using xapian 1.2.22) Some of this is pretty much expected, though other parts I don't
2020 Oct 21
2
xapian-check sorted order error
Hi, We were running xapian-check on one of our Xapian indexes and it returns the following error: position: baseB blocksize=8K items=809896869 lastblock=2090419 revision=3161 levels=3 root=2084903 Failed to check B-tree: DatabaseError: Items not in sorted order The other tables verify without issue. It looks like our oldest backup of this database (a month old) has the same issue. Searching and
2017 Dec 08
2
xapian 1.4 performance issue
Olly Betts writes: > On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > > Recoll builds snippets by partially reconstructing documents out of index > > contents. > > > [...] > > > > The specific operation which has become slow is opening many term position > > lists, each quite short. > > The difference will actually
2017 Dec 07
0
xapian 1.4 performance issue
On Thu, Dec 07, 2017 at 10:29:09AM +0100, Jean-Francois Dockes wrote: > Recoll builds snippets by partially reconstructing documents out of index > contents. > [...] > > The specific operation which has become slow is opening many term position > lists, each quite short. The difference will actually be chert vs glass, rather than 1.2 vs 1.4 as such (glass is the new backend in
2018 Sep 21
1
How to make database build threaded?
On 21/09/2018 at 08:03, Olly Betts wrote: > > You can then either just search those databases together or use > Xapian::Database::compact() (or the xapian-compact command line tool) to > merge the databases once built. > just my 2 cents tips: a commit() to the database is needed before compact() otherwise every attempt to search fails. This on GNU Linux Debian 9.5 Xapian:
2024 Apr 22
2
How to use Xapian Omega directly (i.e., without using `recoll` and `xapiandb`) ... Full Set Of Questions Below:
Dear senior ML members and developers of Xapian Omega, Mr. Olly has helped me cross the bump of the initial learning curve. (ref: https://lists.xapian.org/pipermail/xapian-discuss/2024-April/010034.html) How can I use Xapian Omega directly (i.e., without using `recoll` and `xapiandb`) to index a directory of text files with all strings greater than 3 characters, to create an index text file
2024 Apr 17
1
Is there a front-end for using xapian-omega rather than the terminal? Could a Xapian database be accessed from web-browsers?
Hello friends, I have installed xapian-omega for my system installed from Official Debian GNU/Linux Live 11.6.0 lxde 2022-12-17T11:46, It is an old edition but I am okay with it, I tasted the power of xapian while using recoll. I read that xapian is a powerful database building package for character strings, and indexes a directory of files really well. I want to use xapian-omega as a customisable
2017 Mar 02
2
errors on rebuild
Hi Olly, Thanks for the detailed response. I hadn’t realized there was a new xapian haystack backend. I’m going to try that but I have some upgrades to do first. Django 1.8, etc. Thanks, Ryan > On Feb 28, 2017, at 3:40 PM, Olly Betts <olly at survex.com> wrote: > > On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote: >> I am trying to rebuild an index of 2+
2016 Jul 06
2
Xapian 1.4.0 released
I have installed the new Xapian 1.4.0 , during the installation, I haven't seen any problems, however, when I execute commands quest and delve I get different versions, and my Perl-based searches return Exception: Couldn't detect type of database ... and what are these glass things in the index directories? There is a no new version of Perl Search::Xapian. $ quest -version quest -
2018 Mar 05
2
Xapian 1.4.5 "Db block overwritten - are there multiple writers?" with Glass
Hello, Since the migration to xapian 1.4.X and the migration to Glass, i've recurrent database corruption errors. I've to reindex my database to make it works again. > DatabaseCorruptError: Expected block 200713 to be level 2, not 1 Theses problems always occur after this error: > xapian.DatabaseCorruptError: Db block overwritten - are there multiple writers? I've
2018 Jul 02
2
Is there a large variance in xapian searching?
Dear XAPIAN developers, I was using xapian to index large than 13 million document about Q & A(Quora similarly). I will share some performance data about indexing and searching, and I will seek some help for improving performance of searching. My computer has 8 i7 at 3.4G CPU and 16G memory, ubuntu 16.04. Dataset include about 13M document, each document will be cut into 35 term(Chinese
2009 Apr 12
2
Indexing speed benchmark - Xapian, Solr
I came across this benchmark between Xapian & Solr: http://www.anur.ag/blog/2009/03/xapian-and-solr/ According to the benchmark, a doc set that took Solr 34 min to index took Xapian 7 hours. Solr's index is also much smaller - 2.5GB to Xapian's 8.9GB. I'm new to Xapian. Just wondering if results like these are typical? Is indexing speed & size a known issue in Xapian? Or is
2019 Jul 09
2
Transitioning notmuch/Xapian from 32-bit to 64-bit system
Hi! Suppose you have a huge notmuch/Xapian database, built on a 32-bit system (well, actually on x86_64-pc-linux-gnu, but using a years old 32-bit notmuch binary; notmuch 0.9, Xapian 1.2.21 -- don't laugh), and suppose you're finally going to update that years old notmuch installation (release by release, forward-porting a bunch of patches). Naturally, I'd now do a native 64-bit
2019 Jan 31
4
Amount of writes during index creation
Olly Betts writes: > On Mon, Jan 21, 2019 at 03:25:01PM +0100, Jean-Francois Dockes wrote: > > I have had a problem report from a Recoll user about the amount of writes > > during index creation. > > > > https://opensourceprojects.eu/p/recoll1/tickets/67/ > > > > The issue is that the index is on SSD and that the amount of writes is > >