[Xapian-devel] goodbye QuartzBufferedTable

[Olly Betts]

On Thu, Aug 12, 2004 at 10:32:07PM +0200, Arjen van der Meijden wrote:
> On 12-8-2004 19:47, Olly Betts wrote:
> 
> >On Wed, Aug 11, 2004 at 10:00:20PM +0100, Olly Betts wrote:
> >
> >>Once this test has completed, I'll try upping the number of documents
> >>per batch and see if that improves the rate.
> >
> >Upping the number of documents per batch to 2000 increases the
> >throughput quite dramatically - once things get going the gain is
> >50-60%, and it appears to be doing relatively better as the database
> >size increases (currently this run has done about 700K documents).
> 
> That sounds very good. Let me know whenever that is test-ready and you'd 
> like a/another test on some of (our) real-life data :)

Pretty much now I think.  I should add something so the batch size can
be set without recompiling though.

> Does upping the limit higher, result into more increased throughput?

I don't know - I'm letting each build run for a while to see how well
the changes scale.  So far I've only tried 1000 and 2000, but I'll keep
going to see roughly what is optimal.  I suspect it'll depend on the
data and the hardware (RAM size in particular).  Sadly I doubt it'll be
possible to make it self-tune well.

Cheers,
    Olly