[Xapian-devel] goodbye QuartzBufferedTable

[Arjen van der Meijden]

On 13-8-2004 19:19, Olly Betts wrote:
> On Fri, Aug 13, 2004 at 12:54:40PM +0100, Olly Betts wrote:
> 
>>You should find the memory usage is a lot lower now.  Before we were
>>buffering changes to the tables in memory, which used a lot of memory.
>>Now we just update the Btree and leave it to the OS to cache blocks
>>from the Btree which appears to be a better use of the memory.
> 
> As a data point, my indexer (a custom one, but of similar complexity to
> scriptindex) seems to level off at around 60MB with CVS HEAD.

I ran a small test, using an existing database (915788 documents, 13GB). 
The test consisted of 1824 documents, totalling to 103MB of textual data 
for scriptindex.

The normal 0.8.1:
real    54m31.446s
user    7m44.730s
sys     0m54.070s

The cvs-head:
real    23m17.052s
user    6m1.310s
sys     0m13.110s

The first one consumed up to 521MB of memory (480 resident), while the 
second only used 85MB, that alone makes up for quite a performance boost 
I suppose... (the machine only has 1GB of memory). The second was set to 
flush after 5000 documents (i.e. after all the work was done).

I haven't taken time to check the "dangerous" patch yet, but I'll have a 
look into that (will it somehow be available in the next release?).

Best regards,

Arjen van der Meijden