[Xapian-discuss] Search performance issues and profiling/debugging

[Olly Betts]

On Thu, Oct 25, 2007 at 08:27:33PM +0200, Ron Kass wrote:
> We ran the same test with databases 2,4,6,8 and 10 on a:
>    Dell 2950,
>        CentOS 5.0 64bit
>        Two quad-Xeon CPUs (8 cores total)
>        16GB mem
>        6 disks in raid 5.
>    (a nice and strong machine overall)
> 
> While speed was better in most cases, it was still long for repeated 
> searches.. around 0.2-0.3 seconds for sequential searches (what was 
> around 1 second on the other machine).
> 
> Notes:
>    * There is mysql server running on that machine, but its not doing much.

Hmm, oprofile shows 25-50% of the time is spent in mysqld...

Otherwise, a lot of the time in Xapian seems to spent unpacking posting
lists in flint.  That's not terribly suprising, but it might be
worthwhile looking at the parts of that code which show up in these
profiles to see if they can be optimised.

It would be interesting to know if you get similar profiles with the
bug in BM25Weight fixed (the one which gives varying weights when k2 is
non-zero).  That was fixed in SVN after 1.0.4.

Cheers,
    Olly