[Xapian-devel] Memory requirements for huge queries

[Olly Betts]

On Wed, Jan 05, 2011 at 02:04:53PM +0000, Charlie Hull wrote:
> This turned out to be a slight red herring, as there was an extra bit of  
> cacheing implemented that pushed up memory requirements...we're now  
> seeing around 350MB for 6500 terms.

That's about 55KB per term - each term needs a cursor on the B-tree which
needs at least block_size * number_of_btree_levels, so 55KB isn't totally
implausible.

This could probably be improved, for example the B-tree manager used to
share the upper level blocks in the B-tree between cursors, but keeping
track of this was complex and had subtle bugs, so we gave up on it, but
there may be a simpler way to achieve the same effect.

Cheers,
    Olly