[Xapian-devel] Memory requirements for huge queries

[Charlie Hull]

Hi all,

We've had a customer report some issues recently which turned out to be 
memory related, which we thought might be of interest.

They're running Xapian 1.0.11 on Windows Server 2003 using the Python 
2.4 bindings (yes, I know they could do with some upgrades :-).

They have a client who was sending really, really long queries via their 
system to Xapian - of the order of 6500 terms, with some OP_PHRASE, 
OP_OR and OP_AND operators. As these were being processed by Xapian we 
were seeing peak heap memory requirements of over 2GB - and this was 
causing allocation exceptions elsewhere and bringing the system down, as 
the machine only has 2GB available.

Manually chopping the query roughly in half shows a peak memory 
requirement of 1.5GB, which was survivable in this case - and the memory 
requirement then goes down again indicating no actual memory leaks. 
We're going to implement a maximum query length to protect against 
future problems, but we thought people might be interested in this 
admittedly unlikely scenario. I wonder if it might be easy to graph 
query length against memory required?

Cheers

Charlie
Flax