Kyle J. Ginavan
2009-Jul-28 17:45 UTC
[Xapian-devel] question and potential consulting gig
we are looking to enhance our projects search. we only need searching of database tables. we are currently using postgresql's full text search. from your experience, how long does xapian index take to add delta's to results? on your site you advertise "Allows simultaneous update and searching. New documents become searchable right away." what does right away mean? like less than a minute, 5 minutes (given unlimited hardware resources)? is there db size where adding delta's is slow? if someone can confirm this, i would like to discuss a potential consulting job with them. cheers, Kyle J. Ginavan
On Tue, Jul 28, 2009 at 12:45:33PM -0500, Kyle J. Ginavan wrote:> we are looking to enhance our projects search. we only need searching > of database tables. we are currently using postgresql's full text > search. > > from your experience, how long does xapian index take to add delta's > to results? on your site you advertise "Allows simultaneous update > and searching. New documents become searchable right away." what does > right away mean? like less than a minute, 5 minutes (given unlimited > hardware resources)? is there db size where adding delta's is slow?It going to be rather dependent on the data and the hardware, but to give you an idea, you can add a single document to a database of 600,000 wikipedia articles in 6 seconds elapsed time. Batching is much more efficient than adding single documents - for the same database, you can add a batch of 24 documents in 27 seconds. If you really have unlimited hardware resources, you should be able to do much better (e.g. using SSDs instead of hard disks). The main factors here are going to be the speed of the disk subsystem and having enough RAM to keep the parts of the Xapian database being touched in the VM cache.> if someone can confirm this, i would like to discuss a potential > consulting job with them.Feel free to email me off-list to discuss. Cheers, Olly