[Xapian-discuss] set_cutoff <percent_cutoff> [<weight_cutoff>]

[Kevin Duraj]

Olly,

I want the top speed during indexing and searches, and I do not care about
smallest database. I think most of users feel the same. If "gzip -9" makes
the indexing slightly slower, remove it. *smile* :-)

Thanks,
Kevin

On 5/7/07, Olly Betts <olly at survex.com> wrote:
>
> On Mon, May 07, 2007 at 11:27:24AM -0700, Kevin Duraj wrote:
> > >I install Xapian 0.9.99 svn8485 and was prompt to convert flint
> > >database version 200506110 to 200704230.  I re-indexed database and
> > >indexing seems to run slower. I am concerned because what used to take
> 15
> > >mins to index now takes 45 mins. Do you think that Omega could miss
> reading
> > >environment variable XAPIAN_FLUSH_THRESHOLD ?
> >
> > Opps! ... I've got lot more documents ... that would explain ... :-)
>
> Indeed!
>
> However, note that SVN HEAD now performs zlib compression of tags in the
> record and termlist tables, so it is likely to be a little slower than
> flint in 0.9.X.  Or at least more CPU intensive, which could perhaps
> turn out faster if you're badly I/O bound.
>
> Currently it uses the equivalent of "gzip -9" and tries to compress any
> tag of 5 or more bytes (the shortest zlib can ever compress I found).
> This should give the smallest database, but I suspect there's a
> sweetspot which trades off a small increase in size for much reduced CPU
> usage.  But that tuning can wait for 1.0.X.
>
> Cheers,
>     Olly
>



-- 
Kevin Duraj
http://myhealthcare.com