i used flint as my backend database. and i add sleep in function of FlintTable::add and FlintTable::del to reduce cpu usage. and it works in when call add_document.the cpu usage below 10%. but when i insert 15k documents, it will get very high cpu usage, will have many times more than 40% in two minutes. what does it do in that time. adjust the index? and how can reduce the cpu usage. where can i add sleep to reduce the usage in that two minutes and in the time of commit_transaction 2009-9-2 ouwind
On Wed, Sep 02, 2009 at 03:54:13PM +0800, ouwind wrote:> i used flint as my backend database. and i add sleep in function of > FlintTable::add and FlintTable::del to reduce cpu usage. and it works > in when call add_document.the cpu usage below 10%. but when i insert > 15k documents, it will get very high cpu usage, will have many times > more than 40% in two minutes. what does it do in that time. adjust the > index? and how can reduce the cpu usage. where can i add sleep to > reduce the usage in that two minutes and in the time of > commit_transactionI'm not sure this is a sensible approach. There are many places where CPU time could be spent, and the only way to reliably find them is to spend a lot of time profiling. You're bound to end up with a lot of calls to sleep, and upgrading to a new Xapian release will often be painful, both because your changes will often conflict with changes in the new release, and because the places where CPU time is spent may change. Why not work with the facilities the OS provides? If this is Unix, then use "nice -n19" to tell the scheduler to give least priority to the indexing process. Then other processes will get all the CPU time they want, and the indexer will use as much of the remaining CPU time as it wants. If you have a recent Linux version, "ionice -c3" can do similar things for I/O prioritising. Cheers, Olly
my os is windows. there are a thread used to insert document into xapian, i reduce the priority of the thread to below normal, but i still get high usage. On Wed, Sep 02, 2009 at 12:10:12PM +0100, Olly Betts wrote:> Why not work with the facilities the OS provides? > > If this is Unix, then use "nice -n19" to tell the scheduler to give > least priority to the indexing process. Then other processes will get > all the CPU time they want, and the indexer will use as much of the > remaining CPU time as it wants.Recent Linux (>= 2.6.23) also has "chrt --idle". I've not compared, but it's probably slightly more effective at achieving the above.> If you have a recent Linux version, "ionice -c3" can do similar things > for I/O prioritising.Cheers, Olly
thank u ,i try it to see the effect On Thu, Sep 03, 2009 at 09:12:14AM +0800, ouwind wrote:> my os is windows. there are a thread used to insert document into > xapian, i reduce the priority of the thread to below normal, but i > still get high usage.A quick google suggests THREAD_PRIORITY_IDLE is what you want there, rather than just reducing the priority. High CPU usage is still to be expected, but not to the detriment of other processes. So if another process is CPU bound, you'd expect the indexer to use close to 0%, but if the other processes would only be using 25% CPU, the indexer would be expected to use up to 75%. If this really isn't working, I'm not going to be able to help as Microsoft stuff isn't my area of expertise. If nobody else on the list knows, you could try an OS-specific list/forum/irc channel/etc. If you really want to stick to your original plan, you're going to have to profile to find where CPU time is actually spent (as I mentioned before). Cheers, Olly