[Xapian-discuss] Xapian performance on gmane.org compared

[Arjen van der Meijden]

On 27-8-2009 17:34 Henry wrote:
> [co AND uk] returns double the results, at less than half the speed (15s).

That is actually not very fast, given that the database fits in memory 
and isn't very large. Although we have a tendency to over-specify our 
servers, so I actually don't have a very good base of reverence.

> Out of interest, do you recall what improvement you saw with phrase 
> queries using SSD compared to normal IO?

Our compacted database is about 20GB. The old machine had 8GB ram, 2 
single-core 3.8Ghz Xeon cpu's (from the pentium 4 era), and a raid5 of 
5+1 10k rpm disks (one hotspare). Our new machine has 2 nehalem's 
2.93Ghz quad-core, 24GB ram and 4 ssd's in raid 5 (no hotspare).
I have been able to build phrase queries that would take several minutes 
on the old machine, especially the first time they ran, now I have a 
hard time getting more than a few seconds...

A query like ["windows 7"] took at a good moment 0.7 second, but went up 
to take up to about 9 seconds with the latest Xapian and a fresh 
database back in april 2009. Now it is completed within 0.2 seconds. But 
its a bit hard to compare, we replaced the system that is on all fronts 
much faster and we also reindexed the database on a different file 
system and with a different compaction block size now.
The average of all returned search-pages went down from about 0.35 
seconds to 0.18 seconds, and that still includes sending the results 
internally over the network and doing some processing with those results 
in the webservers.

Best regards,

Arjen