[Xapian-discuss] set_cutoff <percent_cutoff> [<weight_cutoff>]

[Olly Betts]

On Mon, May 07, 2007 at 11:27:24AM -0700, Kevin Duraj wrote:
> >I install Xapian 0.9.99 svn8485 and was prompt to convert flint
> >database version 200506110 to 200704230.  I re-indexed database and
> >indexing seems to run slower. I am concerned because what used to take 15
> >mins to index now takes 45 mins. Do you think that Omega could miss reading
> >environment variable XAPIAN_FLUSH_THRESHOLD ?
>
> Opps! ... I've got lot more documents ... that would explain ... :-)

Indeed!

However, note that SVN HEAD now performs zlib compression of tags in the
record and termlist tables, so it is likely to be a little slower than
flint in 0.9.X.  Or at least more CPU intensive, which could perhaps
turn out faster if you're badly I/O bound.

Currently it uses the equivalent of "gzip -9" and tries to compress any
tag of 5 or more bytes (the shortest zlib can ever compress I found).
This should give the smallest database, but I suspect there's a
sweetspot which trades off a small increase in size for much reduced CPU
usage.  But that tuning can wait for 1.0.X.

Cheers,
    Olly