[Xapian-discuss] how can i reduce cpu usage

[Olly Betts]

On Wed, Sep 02, 2009 at 03:54:13PM +0800, ouwind wrote:
> i used flint as my backend database. and i add sleep in function of
> FlintTable::add and FlintTable::del to reduce cpu usage. and it works
> in when call add_document.the cpu usage below 10%. but when i insert
> 15k documents, it will get very high cpu usage, will have many times
> more than 40% in two minutes. what does it do in that time. adjust the
> index? and how can reduce the cpu usage. where can i add sleep to
> reduce the usage in that two minutes and in the time of
> commit_transaction

I'm not sure this is a sensible approach.  There are many places where
CPU time could be spent, and the only way to reliably find them is to
spend a lot of time profiling.  You're bound to end up with a lot of
calls to sleep, and upgrading to a new Xapian release will often be
painful, both because your changes will often conflict with changes
in the new release, and because the places where CPU time is spent
may change.

Why not work with the facilities the OS provides?

If this is Unix, then use "nice -n19" to tell the scheduler to give
least priority to the indexing process.  Then other processes will get
all the CPU time they want, and the indexer will use as much of the
remaining CPU time as it wants.

If you have a recent Linux version, "ionice -c3" can do similar things
for I/O prioritising.

Cheers,
    Olly