Hello everyone, I have some suspicious slowness with xapian DB workings. I divided xapian DBs by month to minimize search time and updating DB with new data. But I found that even month DB do not allow to search data very fast as it supposed to do. So I separated last day DB from month DB and update it in real-time (with delay 10 secs after each loop). Idea was in using that last day DB with month DB for search and merge last day with month DB at the beginning of every day. I found that idea here. But that merging process took over 4 hours which I suppose extremely slow. I used xapian-compact utility in this way: xapian-compact monthDB lastDayDB updatedMonthDB Month DB contains about 15M docs (about 60 GB), lastDayDB contains about 500K docs. So I had to replacing merging process with updating month DB with the same data that I use for last day DB which is not right, I think. Now I am thinking to divide current month DBinto last day, last week DBs because search usually goes about 3 minutes in month DB. That part I think is very uncomfortable too. Could anyone give me a hints how to speed up the search and merge processes. I have powerful hardware and this still upset me. I use the same server for search and updating DB since I am limited with one server for xapian index. Thanks, Vaso.
On Fri, May 30, 2008 at 01:25:11PM +0700, Vasiliy Sergeev wrote:> I divided xapian DBs by month to minimize search time and updating > DB with new data. But I found that even month DB do not allow to > search data very fast as it supposed to do.My guess (without any further information) is that you're hitting maximum spindle throughput: ie your disks can't move the data around fast enough, and you don't have enough memory to buffer the bits of the database needed for your search profile. Your operating system will provide tools to shore up or disprove this analysis (iostat, vmstat et al on a Unix, for instance). J -- /--------------------------------------------------------------------------\ James Aylett xapian.org james at tartarus.org uncertaintydivision.org