[Xapian-discuss] Xapian performance testing

[Arjen van der Meijden]

On 5-11-2007 14:40, Olly Betts wrote:
> BTW, I have implemented the hoisting of the positional information
> checking part of NEAR and PHRASE, so that the "AND" inside can be
> merged with other AND and FILTER operations.  This gave a big
> performance boost to the slow queries (~50% saving in time just
> from this one change) and a good boost to the other queries (~25%
> saving from just this change).
> 
> This optimisation and all the earlier ones are in 1.0.4, so once you
> upgrade to that, it would be interesting to see what the slow query log
> looks like with these new optimisations in place.

We finally found time to upgrade our 0.9.8 to 1.0.5 and reindex the 
whole database. The results so far are quite good.
When taking the daily average of our forum search result page it went 
down from about 0.55 seconds to about 0.38 seconds.
Looking at the log-files, the slow query log file (queries taking more 
than 2 seconds) dramatically reduced in size. Prior to the update we had 
3190 and 3112 lines in a week and now in the latest week it had only 863 
lines.

The slowest queries seem to be the single-term phrases with a single 
character attached to a common word like 'd-link', 's-video' and 
variants on that with only a few additional terms. As expected, I don't 
see any version numbers anymore in the slow query log.

Best regards,

Arjen van der Meijden
Tweakers.net