[Xapian-discuss] Search performance issues and profiling/debugging

[James Aylett]

On Wed, Oct 24, 2007 at 04:04:22PM +0200, Ron Kass wrote:

> Although we should never rule out something completely without checking, 
> I believe quite strongly that the issues we are seeing are not coming 
> from Xen, as per this instance it is a regular dedicated Linux (centos 
> 5) machine and the resources are fully dedicated to it.

It seems to me that there are two distinct problems. You have some
queries that are underperforming, which with some profiling will
expose either something unusual about your database or code, or a
bottleneck or optimisation problem in Xapian.

The other is the variation. I agree with Chris that adding Xen into
the mix is complicating matters considerably. Things like IO
scheduling, for instance, become harder in even the best
virtualisation systems. It's bad enough that a single instance of an
OS can suddenly start doing things you don't expect, even with no
other significant userspace clients :-/

Out of interest, are your figures averages of multiple runs? If not,
I'd be interested in seeing 1st, 2nd and 3rd query times (broken down
as Olly suggests), but with mean & sd over say 100 runs.

(Apologies if you have done that - I've been trying to follow this
thread closely, but an explosion of posts has combined with a busy
period at my end :-)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org