[Xapian-discuss] search queries with less than 3 characters, memory goes nuts

[chris]

> On Sat, Aug 15, 2009 at 12:58:53PM +0200, chris wrote:
> > So my questions are:
> > - why does xapian use countless gigabytes of ram if i feed it such
> > a query?
> 
> I've never seen it do so before.

I guessed so, just didn't know where to turn first.

> 
> > - is there a need to clean the query before? i mean, could someone
> > do something nasty with it? (except the usual html-security things,
> >   which we take care of by escaping the query before display)
> 
> There shouldn't be a need.

Very well

> 
> > - what can i do to prevent this? 
> 
> My guess is that acts_as_xapian is asking Xapian to return all
> possible matches, is getting a few million, and is storing them in a
> space-inefficient way.
> 
> The code here seems to show @limit defaults to "-1" which I assume
> means "maximum unsigned integer" by the time Xapian sees it:
> 
> http://github.com/Overbryd/acts_as_xapian/blob/dc3517c66b18dbf66733aac3ba436c7bf4ffcab8/lib/acts_as_xapian.rb

I'm overriding it with my own per-page limit and by watching the logs
it seems to accept this, but i'll investigate deeper.

> 
> It would be useful to narrow down which layer is causing this.  Can
> you try running some of these "bad" queries without the Ruby layers
> involved (examples/quest in xapian-core provides an easy way to run a
> query against a database).
> 
> If that works OK, try it from just using the Ruby bindings (without
> acts_as_xapian) - you may find examples/simplesearch.rb useful for
> that. 

Ok, will do.

> If the problem is in acts_as_xapian, you'll need to talk to its
> developers, or just pass a sane limit giving the number of matches you
> actually want.  It's a good idea to do that anyway since asking for
> all possible matches will disable various matcher optimisations and
> slow down searches.

After reading your answer i also suspect acts_as_xapian as the
problem, i just wanted to be sure that i did not make a stupid
mistake, thanks for the help.

Greets, Chris