[Xapian-discuss] Re Phrase Tuning.

[Olly Betts]

On Mon, Nov 07, 2005 at 05:33:32AM -0500, tech at dbx.co.uk wrote:
> I have given flint a go -and it is generally much quicker (once it has
> loaded the cache -that process still takes minutes). Flint actually seems
> tro do some caching whereas quartz just seemed to hit the disc constantly.

It's probably just that flint has a smaller working set so you can cache
enough.

> Generally I've been running it on a machine with 5G of memory -but it has
> to contend with other processes for resources, I have also run it on its
> own on a 1G system -but quartz was atrocious on this (I haven't tried flint
> yet).
> 
> If the management agrees then I'll probably try splinting the index (now
> 12-13G) over 2 machines and running flint

5G of RAM should really be ample for caching enough of a 12-13G
database to give good performance.  I've studied this in the past and
found that you could have less than 5% cached and still get good
performance.  For example, the gmane search database is 53G on a machine
with 3G of RAM.  That's not necessarily directly comparable to your
situation since the gmane database doesn't currently have positional
information, but it should give an idea...

If you're able to get a second box, I'd first try devoting one to the
search and one to the other tasks rather than splitting the search.

> -obviously the fact that it's beta will give them a bit of a problem,
> but it looks like it's that or nothing (or Lucene).

I wouldn't call it beta.

The main difference between Flint and Quartz is really that I have no
qualms about breaking compatibility in the Flint database format.

The Quartz code has been in use for several years, but Flint currently
reuses a lot of Quartz code, which I'm replacing piece by piece and
Xapian has a pretty extensive testsuite.  There are also at least 3
major installations using Flint (include one I take care of), so
breakage should really be pretty unlikely.  

But of course neither Flint nor Quartz (nor Lucene for that matter) come
with any warranty.  If the search features are sufficiently important to
your business that people get nervous about that, you need to address
the risks somehow, for example by performing your own integration
testing before upgrading live installations to a new version.

Cheers,
    Olly