[Xapian-discuss] Benchmark (index, search) on 20 million docs Xapian/Linux vs. Lucene/Windows

[James Aylett]

On Tue, Feb 13, 2007 at 09:57:32AM +0000, Olly Betts wrote:

> > >Result: Xapian/Linux showed to be 35% faster
> > >then Lucene/Windows on searching.
> > 
> > That's the typical Windows overhead for finding and opening files on 
> > windows compared to linux.
> 
> While I can see indexing may involve opening a lot of files, searching a
> database opens relatively few.  It's about 11 for Xapian.  I don't know
> how many it would be for Lucene, but it seems unlikely it's many more,
> so you'd need a really large overhead per file find/open to incur a 35%
> overhead over the total time taken to search a database of this size.

Nonetheless, it would be interesting to see Lucene/Linux on the same
hardware and OS install, for comparison. It may make a difference on
Windows which install of the OS you're using, as (guessing here, but
with I hope some justification) the disk optimisations around
read-ahead may vary. There's a tunable somewhere that affects these
things. Not sure if Lucene/Windows counts as a foreground or
background app in terms of the performance boost tunable, either (or
what that was set to).

> > ReiserFS or anything faster than ext3 would do a bit better.
> 
> Do you have any benchmark results to support choosing reiserfs for this
> use?

I'd be very interested to see ext3 vs JFS vs XFS vs ReiserFS figures,
all on the same spec platters.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org