search for: xapiandb

Olly Betts <olly <at> survex.com> writes: > > You could try: > > delve -t '' ./xapiandb > > That will list the document lengths, so you can see if document 6 is in > that list or not. I am the recoll user mentioned in the first post above. I still have a copy of the (potentially) corrupted index and I did the requested testing. I ran delve -t '' ./xapiandb on the...

...1 results :2:../rcldb/rclquery.cpp:392:enquire->get_mset: exception: Document 6 not found I then went through and checked as above (after installing xapian-tools). I ran the xapian-check on both external indices and both had no problems. I then ran "delve -t term ./xapiandb" and found a long list of IDs, one of which was 6. I then ran "delve -r 6 ./xapiandb" and got a long list of terms, which included 'term' and seemed to be reasonable for a document I then ran "delve -r 6 ./xapiandb -d" and got the following: Data fo...

...r-xr-x 1 root root 30728315 Apr 12 10:43 /usr/lib/libxapian-1.3.so.6 ************************* 452.68user 124.94system 4:42.27elapsed 204%CPU (0avgtext+0avgdata 1055204maxresident)k 0inputs+21046192outputs (0major+41137071minor)pagefaults 0swaps ************************* 793244 /home/dockes/.recoll/xapiandb total 793240 -rw-r--r-- 1 dockes dockes 24150016 Apr 12 10:47 docdata.glass -rw-r--r-- 1 dockes dockes 0 Apr 12 10:47 flintlock -rw-r--r-- 1 dockes dockes 130 Apr 12 10:47 iamglass -rw-r--r-- 1 dockes dockes 577527808 Apr 12 10:47 position.glass -rw-r--r-- 1 dockes dockes 120905728 A...

Dear senior ML members and developers of Xapian Omega, Mr. Olly has helped me cross the bump of the initial learning curve. (ref: https://lists.xapian.org/pipermail/xapian-discuss/2024-April/010034.html) How can I use Xapian Omega directly (i.e., without using `recoll` and `xapiandb`) to index a directory of text files with all strings greater than 3 characters, to create an index text file typically occurs in the End of a Book, with location in specific files, without using Recoll database? I want to create an extensive list first with xapian omega, then have the list post-pr...

...ng deliberate plan or careless inconsistency...) We probably ought to cross-check the two - that shouldn't be costly to do. > This is with Xapian 1.2.16 My guess is that the corruption is caused by the same bug as #645, which was fixed in 1.2.21. > I then ran "delve -t term ./xapiandb" and found a long list of IDs, one of > which was 6. I then ran "delve -r 6 ./xapiandb" and got a long list of > terms, which included 'term' and seemed to be reasonable for a document I > then ran "delve -r 6 ./xapiandb -d" and got the followin...

Hi, I'm using xapian(1.2.2) python interface to connect remote xapian-tcpserv daemon. I'm starting the daemon as given below: xapian-tcpsrv --port 9100 --interface 127.0.0.1 --writable ./xapiandb/ and in script.py: import xapian as xa db = xa.remote_open_writable('localhost', 9100) # other stuff # ... script is a long running program, and when i try to run another instance, i get : xapian.DatabaseLockError: REMOTE:Unable to get write lock on /home/hinoglu/xap...

On Sun, Jan 10, 2016 at 02:53:14AM +0000, Bob Cargill wrote: > I am the recoll user mentioned in the first post above. I still have a copy > of the (potentially) corrupted index and I did the requested testing. > > I ran delve -t '' ./xapiandb on the index and it returned a very long list > of document IDs, separated by spaces. I than ran delve -t '' ./xapiandb | > grep " 6 " and it returned nothing. > > So, document 6 was not in the list. > > There were other documents missing from the index as we...

Olly Betts writes: > On Sun, Apr 10, 2016 at 04:47:01PM +0200, Jean-Francois Dockes wrote: > > Some might notice the 50% index size increase. Excessive index size is > > already one relatively rare, but recurring complaint. Except if I did > > something wrong: I'm actually quite surprised by it. > > Did you try compacting the resulting databases? > >

A Recoll user gets the following message while indexing: "Attempted to delete or modify an entry in a non-existent posting list for #bannerholder" The exception happens during a commit call. Xapian version 1.4.11, Debian Buster A little more detail here: https://opensourceprojects.eu/p/recoll1/tickets/108/ I asked if this was reproducible, and to run the indexing in single-thread

...ure apache to tell it to use this for a URL on > your site. How Mr. Betts? Suppose I have a ready open document text file, a LibreOffice text file. say BettsBio.odt, for which I want to create an index file, say, BettsIndex.txt. Please assume that I have used Recoll on BettsBio.odt to create a xapiandb directory and had the database created there already. How do I configure Apache for the purpose to use a web-browser? How do I write the command line to access the web-browser? How do I have omega create the text database with extensive location info of words or phrases within the doc? Best wishe...

...etts writes: > On Sun, Jan 10, 2016 at 02:53:14AM +0000, Bob Cargill wrote: > > I am the recoll user mentioned in the first post above. I still have a copy > > of the (potentially) corrupted index and I did the requested testing. > > > > I ran delve -t '' ./xapiandb on the index and it returned a very long list > > of document IDs, separated by spaces. I than ran delve -t '' ./xapiandb | > > grep " 6 " and it returned nothing. > > > > So, document 6 was not in the list. > > > > There were other do...

...d post the > > output. > > That's a good thing to check. If xapian-check finds no problems, then > it's presumably just an in-core issue, which points to a Xapian bug or > memory issues. The output of xapian-check follows. Best regards, Jf xapian-check ~/.recoll/xapiandb record: baseB blocksize=8K items=943378 lastblock=85955 revision=6207 levels=2 root=18014 B-tree checked okay record table structure checked OK termlist: baseB blocksize=8K items=1886756 lastblock=417475 revision=6207 levels=3 root=83720 B-tree checked okay termlist table structure checked OK pos...

...news is that the patch above decreases the amount of writing by a significant factor, around 4.5 for the biggest index I tried. The amount of writes is estimated with iostat before/after. The disk has nothing else to do. idxflushmb is the number of megabytes of input text between Xapian commits. xapiandb,kb writes,kb K*1000 sz/w xapian 1.4.5 idxflushmb 200 1544724 6941286 3.62 4.49 3080540 16312960 3.02 5.30 4606060 21054756 2.13 4.57 6123140 33914344 2.24 5.54 7631788 50452348 2.39 6.61 xapian git master latest idxflushmb 200 1402524 1597352 0.96 1.14 2223076 3291588 0....

...> > I asked if this was reproducible, and to run the indexing in single-thread > mode to simplify the situation. It's worth running xapian-check on the database to see what it reports. Also might be interesting to check what the posting list for that term is: xapian-delve ~/.recoll/xapiandb -vv -t '#bannerholder' > I'm not too sure if a Recoll bug could cause this, or if this has to be a > Xapian issue, I can open a ticket if more appropriate. It shouldn't be possible to cause this via valid use of the API, but bugs in the application could - for example a stra...

...> output. > > > > That's a good thing to check. If xapian-check finds no problems, then > > it's presumably just an in-core issue, which points to a Xapian bug or > > memory issues. > > The output of xapian-check follows. > xapian-check ~/.recoll/xapiandb [...] > postlist: > baseB blocksize=8K items=8872525 lastblock=524452 revision=6207 levels=3 root=238 > B-tree checked okay > termfreq 197211 != # of entries 197210 > collfreq 10861536 != sum wdf 10861533 > termfreq 14189 != # of entries 14188 > collfreq 98354 != sum wdf 98344...

...nc tmp.db/position.DB read 1 from tmp.db/position.DB sync tmp.db/termlist.tmp sync tmp.db/termlist.DB read 1 from tmp.db/termlist.DB sync tmp.db/record.tmp sync tmp.db/record.DB read 1 from tmp.db/record.DB > idxflushmb is the number of megabytes of input text between Xapian commits. > > xapiandb,kb writes,kb K*1000 sz/w > > xapian 1.4.5 idxflushmb 200 If you're going to the trouble of profiling, probably best to use the latest release (1.4.5 was released in 2017). > 1544724 6941286 3.62 4.49 > 3080540 16312960 3.02 5.30 > 4606060 21054756 2.13 4.57 > 6123140...

...fice text file. say BettsBio.odt, for which I want to create an > index file, say, BettsIndex.txt. A Xapian index is not a text file, rather it's a directory of binary files, so naming it `.txt` would be confusing. > Please assume that I have used Recoll on BettsBio.odt to create a > xapiandb directory and had the database created there already. > > How do I configure Apache for the purpose to use a web-browser? This isn't really the right place to seek help in setting up a particular webserver, and it's also not really my area of expertise, but the relevant apache docum...

...he amount of writing > by a significant factor, around 4.5 for the biggest index I tried. > > The amount of writes is estimated with iostat before/after. The disk has > nothing else to do. > > idxflushmb is the number of megabytes of input text between Xapian commits. > > xapiandb,kb writes,kb K*1000 sz/w > > xapian 1.4.5 idxflushmb 200 > > 1544724 6941286 3.62 4.49 > 3080540 16312960 3.02 5.30 > 4606060 21054756 2.13 4.57 > 6123140 33914344 2.24 5.54 > 7631788 50452348 2.39 6.61 > > xapian git master latest idxflushmb 200 > > 1402524...

Hi, I have had a problem report from a Recoll user about the amount of writes during index creation. https://opensourceprojects.eu/p/recoll1/tickets/67/ The issue is that the index is on SSD and that the amount of writes is significant compared to the SSD life expectancy (index size > 250 GB). >From the numbers he supplied, it seems to me that the total amount of block writes is roughly

Hi, I have a user reporting the following error during recoll indexing: flush() failed: Db block overwritten - are there multiple writers? "flush() failed" is from recoll, the rest is, I think the text of the Xapian exception. This is with Xapian 1.4.3 on Linux (I asked for more details, should be coming). I don't think that I've ever seen this error, and I also