Xapian discuss - Apr 2005 - Two questions

Hi there,
I have implemented  Xapian with success on two millions html documents and
work very very good! Thanks for this nice software! 

Results are almost alwayse returned in less then 1 second but when I use
set_collapse_key (I really need this)  searches can take up to 10 secondes.
Is there any way to speed up this?

Is it possible to get list of most used words in database?

Thanks!
Roki

-- 
+++ GMX - die erste Adresse für Mail, Message, More +++

10 GB Mailbox, 100 FreeSMS  http://www.gmx.net/de/go/topmail

roki roki wrote:
>Hi there,
>I have implemented  Xapian with success on two millions html documents and
>work very very good! Thanks for this nice software! 
>
>Results are almost alwayse returned in less then 1 second but when I use
>set_collapse_key (I really need this)  searches can take up to 10 secondes.
>Is there any way to speed up this?
>  
>
Because you are collapsing multiple matching documents into 1 hit,
xapian has to find more documents to get the same number of hits.

If you are using omega you could try reducing MIN_HITS (may be called
MINHITS now) which reduces how many results xapian tries to find, but of
course also reduces the accuracy of how many hits xapian thinks there
might be.
>Is it possible to get list of most used words in database?
>
>  
>
I'll let someone else answer this.

Sam
>Thanks!
>Roki
>
>  
>

On Sat, Apr 30, 2005 at 06:31:08PM +0200, roki roki
wrote:
> Is it possible to get list of most used words in database?
Not directly.  You can iterate over all the terms in the database (using
Xapian::Database::allterms_begin() and allterms_end()) and inspect the
term frequencies.  Terms aren't stored sorted or indexed by frequency,
so there isn't a magically more efficient way.

Cheers,
    Olly

On Fri, May 13, 2005 at 10:17:28AM +0200, roki roki
wrote:
> as promised here is test perl script with some data. As you can see I only
> use add_term and replace_document.
Thanks!  That's very much appreciated.
> When I generate first time database with
> this script and then execute quartzcompact I get the following result:
> 
> postlist: Reduced by 49.7817% 912K (1832K -> 920K)
> termlist: Reduced by 39.4366% 224K (568K -> 344K)
> 
> When I execute 5 time this test script and the do quartzcompact I get:
> 
> postlist: Reduced by 74.8908% 2744K (3664K -> 920K)
> termlist: Reduced by 69.7183% 792K (1136K -> 344K)
I'm pretty confident that I can improve things, though it's unlikely to
be an instant fix.
    
> I have also noticed that producing a new database on my 2,5 mil records
data
> starting with 60.000 records per hour but after few hundred thousands it go
> to the 30.000 records per hour which. 
It's not at all suprising that the initial rate isn't sustained.  The
first batch of records flushed can just be plonked straight in, but
subsequent batches need to be merged in.  There's another threshold
when the working set fails to fit entirely in RAM.

You can see some performance graphs for gmane indexing here:

http://www.survex.com/~olly/gmaneindexrate.html

I'm afraid I forget exactly what the 3 colours mean or what the
difference between the 3 graphs is, but you can see that the early
performance does drop off.
> This is on the machine with 1 GB RAM and XAPIAN_FLUSH_THRESHOLD = 20000; 
> Can I improve this speed?
Have you experimented to see if 20000 is the best setting?

Otherwise you can build several smaller databases and then merge them
using quartzcompact.  For details and some figures, see:

http://thread.gmane.org/gmane.comp.search.xapian.general/1462

The changes it talks about are in the 0.9.0 release.

Posting list changes are buffered in RAM before being flushed to disk,
which is currently done using standard STL data structures.  We could
be a lot more compact here by exploiting what we know about the
structure of the data (at least in the common case of adding new
documents), which would greatly increase the amount that could be
buffered in a particular size of spare memory.  This is something I'll
be working on fairly soon.
> P.S. I am sending only to you this mail because file is little to big
That's fine.  I've replied to the list because this is probably of wider
interest.

Cheers,
    Olly