[Xapian-devel] goodbye QuartzBufferedTable
olly at survex.com
Fri Aug 13 00:56:02 BST 2004
On Thu, Aug 12, 2004 at 10:32:07PM +0200, Arjen van der Meijden wrote:
> On 12-8-2004 19:47, Olly Betts wrote:
> >On Wed, Aug 11, 2004 at 10:00:20PM +0100, Olly Betts wrote:
> >>Once this test has completed, I'll try upping the number of documents
> >>per batch and see if that improves the rate.
> >Upping the number of documents per batch to 2000 increases the
> >throughput quite dramatically - once things get going the gain is
> >50-60%, and it appears to be doing relatively better as the database
> >size increases (currently this run has done about 700K documents).
> That sounds very good. Let me know whenever that is test-ready and you'd
> like a/another test on some of (our) real-life data :)
Pretty much now I think. I should add something so the batch size can
be set without recompiling though.
> Does upping the limit higher, result into more increased throughput?
I don't know - I'm letting each build run for a while to see how well
the changes scale. So far I've only tried 1000 and 2000, but I'll keep
going to see roughly what is optimal. I suspect it'll depend on the
data and the hardware (RAM size in particular). Sadly I doubt it'll be
possible to make it self-tune well.
More information about the Xapian-devel