[Xapian-discuss] Xapian performance on gmane.org compared

[Henry]

Quoting "Arjen van der Meijden" <acmmailing at tweakers.net>:
> You could have a look at the size of the result for non-phrased co  
> and uk (i.e. "co AND uk"). We've seen pretty bad performance for  
> some phrase queries in the flint-database, but then our machine used  
> to be io-dependent. This should give you an idea of how many  
> documents are loaded from disk for the initial selection and how  
> fast that goes.
> But since the phrase-query touches another large table, you can't  
> use it as more than a simple base line.

[co AND uk] returns double the results, at less than half the speed (15s).

> 4GB is a "very small" database, i.e. it can fit in a amount of ram  
> that is now becoming common for desktops. How much memory does your  
> search-machine have? If it doesn't have at least 4GB, and you can  
> spare a bit of money, increase it.

It has enough RAM, but your point is taken (the machine is also  
low-level busy with some other stuff which is stealing RAM) - we'll  
eventually scale this up to multiple nodes in a cluster (with a heck  
of a larger database), but I need to gauge how this performs on a  
single machine first to gain confidence that this is going to work on  
a larger scale (taking into account that there is a limit to the  
cluster size).

> If there are no other factors in play, and your query-performance is  
> solely or largely caused by lacking I/O-performance, you could also  
> install a ssd-drive. With our benchmark, we had all phrase-queries  
> turn from io-limited into cpu-limited, simply because both the ram  
> and ssd's in our server just were easily fast enough to keep up.

Yes, this is also an option with us.  ie, rather spend on SSD where it  
makes sense, instead of many more search nodes.

Out of interest, do you recall what improvement you saw with phrase  
queries using SSD compared to normal IO?

Cheers
Henry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: PGP Digital Signature
Url : http://lists.xapian.org/pipermail/xapian-discuss/attachments/20090827/897b1419/attachment.pgp