[Xapian-discuss] Improving indexing speed

[Robert Kaye]

Hi!

After more work I've managed to get Xapian to work better all around  
than our previous text search engine. I've been able to tweak, or work  
around the idiosyncrasies of our data/setup and am getting results I'm  
quite happy with. Big thumbs up to the Xapian dev team!

I often times get rewarded with good chocolate from various corners of  
the world. Do you folks like good chocolate? I can share!

Onward: However, indexing speed is a bit a of a problem for me;  
smaller indexes build faster than the previous system, large indexes  
take about 2-3 times as long.

I noticed disk access is very spikey -- every 3-5 seconds utilization  
goes to 100%. Then there are long periods of 100% disk utilization. My  
CPU is never very busy -- at most I find a 50% - 60% load.  And the  
indexing process only uses about 5% of available RAM. Is there any way  
I can instruct Xapian to use more resources to speed up indexing? My  
index could also be built on a RAM disk -- I suspect that would help,  
but I'm curious as to what the best practices are...

Any tips for speeding up this process would be greatly appreciated!

--

--ruaok      Somewhere in Texas a village is *still* missing its idiot.

Robert Kaye     --     rob at eorbit.net     --    http://mayhem-chaos.net