[Xapian-discuss] Optimization and Load balancing with Xapian

Fri Feb 24 14:20:35 GMT 2006

David Levy wrote:
> Thanks Arjen,
> 
> my hardware is not so bad ;-)
> dual xeon 3.0GHz
> RAM 2Go
> but .... IDE 200Go HD

Its not too bad indeed, but we're running with dual xeon 3.8Ghz, 8GB ram 
and 5x western digital raptor's in raid5. For i/o the latter two make a 
major difference of course.

> I want to test all these on my SCSI HD server too when I have time.
> 
> how many matches do you have in your results in general ? 100 ? 1000 ? 
> more ?

Its a bit time-consuming to browse to the end of the resultset. But when 
sorting on a value it reports there are 17471 results to the query 'cpu 
ram' and the timings below where for the first 100 results.

> how many values do you store in xapian ?

We have 4 different values per document. In your other response I 
noticed you have a 1GB value database... for just 700k documents? That's 
pretty large. Ours is just 133MB uncompacted and 64MB when compacted and 
zlib-compressed!
The 4 values we store are (zero-padded) unix timestamps (32bits), which 
xapian translates to string-values so that's probably just below a 
hundred bytes per document.

> the times i provided are for first run queries of course :)
Well, you miss the positive effects of caching in the runs, which on the 
other hand mask i/o overhead a bit.

Best regards,

Arjen

> On 2/24/06, *Arjen van der Meijden* <acmmailing at tweakers.net 
> <mailto:acmmailing at tweakers.net>> wrote:
> 
>     For our set-up I ran some tests and noticed the sorted-by-value runs
>     were indeed a slower, but not as dramatic as yours. It varied from 2 to
>     8 times as slow (the time Omega reports) than the non-value sorted
>     search. The query 'cpu ram' (wdf resp 80108 and 62900) took 0.107
>     seconds in relevance-sorting and 0.336 when sorting on a date.
> 
>     Of course our hardware is (likely) much faster than yours, but our
>     database is larger too (1.1M documents and 12GB on disk in compacted
>     flint). Are these the initial runs for those queries, or did you rerun
>     the tests a few times, i.e. its unfair to compare a non-cached
>     value-sorted run to a (then) cached relevance-sorted run.
> 
>     Best regards,
> 
>     Arjen
> 
>     David Levy wrote:
>      > Ok for the directory :)
>      >
>      > Here are some results :
>      >
>      > (time /usr/local/lib/omega/bin/omega DB=products P=xxxxxxx SORT=3)
>      >
>      > test paris
>      > real     0m5.130s
>      > user    0m0.035s
>      > sys     0m0.039s
>      >
>      > paris france
>      > real    0m19.902s
>      > user    0m0.049s
>      > sys     0m0.116s
>      >
>      > new york
>      > real    4m10.437s
>      > user     0m0.459s
>      > sys     0m1.601s
>      >
>      > !!!!!!!!!!!!!!!!!!!!!!!!
>      >
>      > that's crazy :)
>      >
>      > maybe because today i don't hae many available RAM (i use most of
>     my RAM
>      > to test RAMDISK for xapian on this server)
>      >
>      > what should i see with this figures ?
>      >
>      > regards
>      >
>      >
>      > On 2/24/06, *Olly Betts* <olly at survex.com
>     <mailto:olly at survex.com> <mailto: olly at survex.com
>     <mailto:olly at survex.com>>> wrote:
>      >
>      >     On Fri, Feb 24, 2006 at 02:35:46PM +0200, David Levy wrote:
>      >      > Yes, I always use Omega PHP5 bindings were not available.
>      >      > Do you think it would be faster with 0.9.3 + php5 bindings ??
>      >
>      >     I'd be suprised if it was.
>      >
>      >      > Thanks for the command, but I can't make it work. In which
>      >     directory should
>      >      > I execute it ?
>      >      > Here is the error :
>      >      >
>      >      > $ time /usr/local/lib/omega/bin/omega DB=products P='test
>     query'
>      >     SORT=1
>      >      > Content-Type: text/html
>      >      >
>      >      > Exception: Couldn't read format template `query' (No such
>     file or
>      >     directory)
>      >
>      >     Where is your omega.conf file?  Where are the omegascript
>     templates?
>      >
>      >     Cheers,
>      >         Olly
>      >
>      >
>      >
>      >
>      > --
>      > David LEVY {selenium}
>      > Website ~ http://www.davidlevy.org
>      > Wishlist Zlio ~ http://david.zlio.com/wishlist
>      > < http://david.zlio.com/wishlist>
>      > Blog ~ http://selenium.blogspot.com
>      >
>      >
>      >
>     ------------------------------------------------------------------------
> 
>      >
>      > _______________________________________________
>      > Xapian-discuss mailing list
>      > Xapian-discuss at lists.xapian.org
>     <mailto:Xapian-discuss at lists.xapian.org>
>      > http://lists.xapian.org/mailman/listinfo/xapian-discuss
> 
> 
> 
> 
> -- 
> David LEVY {selenium}
> Website ~ http://www.davidlevy.org
> Wishlist Zlio ~ http://david.zlio.com/wishlist
> Blog ~ http://selenium.blogspot.com
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xapian-discuss mailing list
> Xapian-discuss at lists.xapian.org
> http://lists.xapian.org/mailman/listinfo/xapian-discuss