[Xapian-devel] Memory requirements for huge queries
Olly Betts
olly at survex.com
Sun Jan 9 09:40:44 GMT 2011
On Wed, Jan 05, 2011 at 02:04:53PM +0000, Charlie Hull wrote:
> This turned out to be a slight red herring, as there was an extra bit of
> cacheing implemented that pushed up memory requirements...we're now
> seeing around 350MB for 6500 terms.
That's about 55KB per term - each term needs a cursor on the B-tree which
needs at least block_size * number_of_btree_levels, so 55KB isn't totally
implausible.
This could probably be improved, for example the B-tree manager used to
share the upper level blocks in the B-tree between cursors, but keeping
track of this was complex and had subtle bugs, so we gave up on it, but
there may be a simpler way to achieve the same effect.
Cheers,
Olly
More information about the Xapian-devel
mailing list