[Xapian-discuss] Improving indexing speed
rob at eorbit.net
Thu Jun 26 19:41:29 BST 2008
After more work I've managed to get Xapian to work better all around
than our previous text search engine. I've been able to tweak, or work
around the idiosyncrasies of our data/setup and am getting results I'm
quite happy with. Big thumbs up to the Xapian dev team!
I often times get rewarded with good chocolate from various corners of
the world. Do you folks like good chocolate? I can share!
Onward: However, indexing speed is a bit a of a problem for me;
smaller indexes build faster than the previous system, large indexes
take about 2-3 times as long.
I noticed disk access is very spikey -- every 3-5 seconds utilization
goes to 100%. Then there are long periods of 100% disk utilization. My
CPU is never very busy -- at most I find a 50% - 60% load. And the
indexing process only uses about 5% of available RAM. Is there any way
I can instruct Xapian to use more resources to speed up indexing? My
index could also be built on a RAM disk -- I suspect that would help,
but I'm curious as to what the best practices are...
Any tips for speeding up this process would be greatly appreciated!
--ruaok Somewhere in Texas a village is *still* missing its idiot.
Robert Kaye -- rob at eorbit.net -- http://mayhem-chaos.net
More information about the Xapian-discuss