[Xapian-discuss] Improving indexing speed

[Robert Kaye]

On Jun 30, 2008, at 11:57 PM, Olly Betts wrote:
> Yes, it's always interesting to hear performance reports.

Ok, I've tinkered with the setup a bit. I've found that if I give  
xapian loads and loads of RAM, it doesn't even get around to using all  
the RAM I give it -- at most each process used 5% of 8G of RAM.

I measured disk access with:

     iostat -x 10 (10 second disk usage average window)

And CPU util with top. I've found:

3 processes: 95% - 96% CPU usage for each process, 40%-60% disk usage
4 processes: 95% - 96% CPU usage for each process, 60%-90% disk usage
5 processes: 92% - 94% CPU usage for each process, 80%-100% disk usage
6 processes: 91% - 93% CPU usage for each process, 100% disk usage  
sustained

It looks like 4 processes is the sweet spot that doesn't utterly slam  
the machine. This is much better than I had anticipated -- well done  
Xapian team!

--

--ruaok      Somewhere in Texas a village is *still* missing its idiot.

Robert Kaye     --     rob at eorbit.net     --    http://mayhem-chaos.net