[Xapian-devel] goodbye QuartzBufferedTable

[Arjen van der Meijden]

On 13-8-2004 1:56, Olly Betts wrote:
> On Thu, Aug 12, 2004 at 10:32:07PM +0200, Arjen van der Meijden wrote:
> 
>>On 12-8-2004 19:47, Olly Betts wrote:
>>That sounds very good. Let me know whenever that is test-ready and you'd 
>>like a/another test on some of (our) real-life data :)
> 
> Pretty much now I think.  I should add something so the batch size can
> be set without recompiling though.

I'll watch the cvs-commits for this. Will you also allow a switch (or an 
environment value or whatever) on scriptindex to adjust this?

>>Does upping the limit higher, result into more increased throughput?
> 
> I don't know - I'm letting each build run for a while to see how well
> the changes scale.  So far I've only tried 1000 and 2000, but I'll keep
> going to see roughly what is optimal.  I suspect it'll depend on the
> data and the hardware (RAM size in particular).  Sadly I doubt it'll be
> possible to make it self-tune well.

Making it runtime/startuptime adjustable will at least allow easier 
searching for semi-optimal values. Finding the real-optimal values will 
probably cost a lot of extra time, while not really improving the 
performance that much. Apart from the fact that it may vary over time; 
database size and structure and data input can differ quite a bit over a 
long index batch.
Currently we allow scriptindex to either run with 1000 documents or a 
set of documents that results in 16MB of data (whichever limit comes 
first) and that makes scriptindex use amounts in the range of 150-250MB 
of ram.

Best regards,

Arjen van der Meijden