similar to: indexing performance

Gentoo Linux 2.6 8 AMD Opteron 64-bit Processors 32GB Memory -------------------------------------------------------------------------------- Environment: ------------------ XAPIAN_FLUSH_THRESHOLD=21000000 XAPIAN_FLUSH_THRESHOLD_LENGTH=16000000 XAPIAN_PREFER_FLINT=True Indexing 20 million documents: --stemmer=none ------------------------------------------- real 79m9.378s user 77m28.696s

There is is bug when setting XAPIAN_FLUSH_THRESHOLD=20000000 When trying for force Xapian flush documents to flush after 20 million documents Xapian ignores the size and flush it after only 10,000 documents. Data captured from delve after 60 seconds interval when has been set as follow: XAPIAN_FLUSH_THRESHOLD=20000000 perl -e ' while(1) { system("delve ."); sleep(60); } '

I'm playing around with a machine that has 2 GB of memory. Indexing about 5GB of data average of 2MB per document. The documents are plain text. I notice the omindex's memory fott print get's biger an bigger then the machine starts to swap and it all slows down to a crawl. In regards to export XAPIAN_FLUSH_THRESHOLD I know the default is 10000 Am I right in saying that for my setup

I'm finding that omindex is consistently ending prematurely when indexing certain files. The last output looks like this: [Entering directory /compounds/Acetic_acid] Indexing "/MATLAB/compounds/Acetic_acid/AACID_50T.TXT" as text/plain ... added. Indexing "/MATLAB/compounds/Acetic_acid/AACID_50T.pdf" as application/pdf ... "pdftotext -enc UTF-8

hi, i use xapian to index a txt file, it's size is 268M. i take each line as a document, and each line has two field like 13445511 | 111115151. the recored size is 10000000. the XAPIAN_FLUSH_THRESHOLD set 1000000. it takes 1026544ms to index the file, it is more slower than lucene. The lucene speed is about 40000 records per second. code: try { Xapian::WritableDatabase

I am create index for some files,in my program,a document is a line in a file. i create index for very lines in a file. is there any method to speed up this ??????

I'm new to Xapian & need some help, many thanks if anyone replies. I did a release build from xapian-core-1.0.7 with VS2008 by using Charlie Hull's makefiles. I'm trying to test-index my dataset -- some 200'000 docs, each document being (on average) 50 bytes long and having 6 words. I tried (a) not to use stemmer, (b) commit_transaction() on every 50/100/etc. docs, (c) not

Hi, Is it possible to perform a search without flushing the index? I've got an application that updates the index every 4 hours but I need to be able to search the new data fairly quickly after the index is updated. The problem revolves around the fact that the update is often much less than 10 000 documents so it isn't being flushed until quite a bit latter. I realise I can do a flush

On Sat, Mar 25, 2017 at 06:36:25PM -0500, Ryan Cross wrote: > After upgrades my stack is now: > > Python 2.7 > Django 1.8 > Haystack 2.6.0 > Xapian 1.4.3. (latest xapian haystack backend with some modifications) > > Using the same rebuild command as below but with —batch-size=50000 > > The issue has now become one of performance. I am indexing 2.2 million >

Running notmuch from git on Debian testing[1] with the mail and database sitting on a ZFS filesystem, adding mail to a new database: > agrajag-testing ~/s/notmuch % ./notmuch new > Found 605510 total files (that's not much mail). > add_file: A Xapian exception occurred36m 37s remaining). > A Xapian exception occurred adding message: Unexpected end of posting list for

I came across this benchmark between Xapian & Solr: http://www.anur.ag/blog/2009/03/xapian-and-solr/ According to the benchmark, a doc set that took Solr 34 min to index took Xapian 7 hours. Solr's index is also much smaller - 2.5GB to Xapian's 8.9GB. I'm new to Xapian. Just wondering if results like these are typical? Is indexing speed & size a known issue in Xapian? Or is

Flint failed to deliver indexing performance to Quartz. I am proposing to remove Flint as default database and place Quartz database back as default. The catch is not that Flint database is smaller and faster during searches then Quartz database as developers were concerning when were measuring and neglecting to measure performance when creating the large indexes. The truth is that Flint

I have my 640GB USB drive connected to a CentOS 5 server and using fdisk, it seems to show a partition of the correct size, /dev/sdd1, but after mounting, the drive shows only 244M size. Does the message received when running fdisk below mean I should reformat this drive, I can't seemed to figure out how to do that with fdisk. Can someone tell me how to reformat, if needed? The drive is empty,

Hi Olly, Thanks for the detailed response. I hadn’t realized there was a new xapian haystack backend. I’m going to try that but I have some upgrades to do first. Django 1.8, etc. Thanks, Ryan > On Feb 28, 2017, at 3:40 PM, Olly Betts <olly at survex.com> wrote: > > On Mon, Feb 27, 2017 at 10:29:46AM -0800, Ryan Cross wrote: >> I am trying to rebuild an index of 2+

Hi Rowland, this is a good question. Does this listing help or answer? root at OMTNDC3:/usr/local/samba/private/sam.ldb.d# ls -al total 93220 Node,CN=Schema,CN=Configuration,DC=omtn,DC=de drwxr-x--- 2 root bind????? 4096 Apr 28 16:03 . drwx------ 7 root root????? 4096 Apr 29 11:25 .. -rw------- 1 root staff 30384128 Apr 29 12:46 'CN=CONFIGURATION,DC=OMTN,DC=DE.ldb' -rw------- 1 root

On 29/04/2020 10:58, Olaf Dreyer via samba-technical wrote: > > I run a setup with two Samba DC (currently samba 4.12.2 on debian 10 > VMs). I started with a Windows 2003 DC but the last Windows DC has > been removed a few years ago from this setup. this is really the wrong list for this, so i have CC'ed the samba mailing list, please reply there. You say you started with Win

Hi, I've a program which: 1. Sets XAPIAN_FLUSH_THRESHOLD=1000 2. Opens a (new) database for write 3. Indexes a few thousand documents 4. Periodically also does queries on the database With 1.1.4, with certain document sets (basically a particular mail folder of mine), Enquire.get_mset() sometimes (but not always) triggers a "RangeError: Value in posting list too

Hello I have some problems to read large data file with R. can someone tell me why running R --visze=30M --nsize=2000k uses in fact 63M? thank you meriema -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the

Greetings All, SuSE 9.1 (64 bit) attempting to set up Wine with an existing Windows 2000K drive. The drive is mounted as vfat on /windows/C, but there are tons of messages about not finding things in "C:\Windows". Do I need to pick through and edit everything (e.g. windows -> winnt) of is there an easier way to do this ? Is there a setup guide for this type of thing ? regards,

I have tried all I can think of to resolve a samba performance problem. Below is a description of the problem. The Red Hat server is actually a higher performance machine than the NT machine, though they are close in capabilities: > When I first installed 5.0 I notices slow samba performance. By slow I > mean around 80K/second doing a normal copy to the server from a Windows > 95