[Xapian-discuss] search queries with less than 3 characters, memory goes nuts
chris at s-4-u.net
Sat Aug 15 15:04:48 BST 2009
> On Sat, Aug 15, 2009 at 12:58:53PM +0200, chris wrote:
> > So my questions are:
> > - why does xapian use countless gigabytes of ram if i feed it such
> > a query?
> I've never seen it do so before.
I guessed so, just didn't know where to turn first.
> > - is there a need to clean the query before? i mean, could someone
> > do something nasty with it? (except the usual html-security things,
> > which we take care of by escaping the query before display)
> There shouldn't be a need.
> > - what can i do to prevent this?
> My guess is that acts_as_xapian is asking Xapian to return all
> possible matches, is getting a few million, and is storing them in a
> space-inefficient way.
> The code here seems to show @limit defaults to "-1" which I assume
> means "maximum unsigned integer" by the time Xapian sees it:
I'm overriding it with my own per-page limit and by watching the logs
it seems to accept this, but i'll investigate deeper.
> It would be useful to narrow down which layer is causing this. Can
> you try running some of these "bad" queries without the Ruby layers
> involved (examples/quest in xapian-core provides an easy way to run a
> query against a database).
> If that works OK, try it from just using the Ruby bindings (without
> acts_as_xapian) - you may find examples/simplesearch.rb useful for
Ok, will do.
> If the problem is in acts_as_xapian, you'll need to talk to its
> developers, or just pass a sane limit giving the number of matches you
> actually want. It's a good idea to do that anyway since asking for
> all possible matches will disable various matcher optimisations and
> slow down searches.
After reading your answer i also suspect acts_as_xapian as the
problem, i just wanted to be sure that i did not make a stupid
mistake, thanks for the help.
More information about the Xapian-discuss