[Xapian-discuss] Slow phrase performance

[Olly Betts]

On Sun, Oct 02, 2011 at 07:43:08AM +0200, Henry C. wrote:
> On Fri, September 30, 2011 18:49, Mark Malloy wrote:
> > I have been unable to find reports of other people experiencing the
> > same problem with poor phrase performance, so I am hoping it's simply something
> > that I'm doing wrong or ineffectively.  Any help would be appreciated.
> 
> Phrase searching is very slow (comparatively) in my experience as well.  Make
> sure your machine can get that whole index into it's cache to eliminate IO to
> maximise performance (ie, if your index is 8GB or 16GB, then try to have at
> least as much RAM, more is better).

You don't need everything cached, as a significant proportion of the
index will never get used in a search on most real world systems
(because nobody will try to search for it).  But certainly adding memory
is a good way to address slow searches if the slowness is down to I/O.

If you're still using 1.0.x (or 1.2.x but with a flint database) then
switching to 1.2.x with its default chert database backend should be
faster for searches using positional data (i.e. phrases and OP_NEAR).

There's a patch in #394 which can make a substantial difference for
slow phrase searches:

http://trac.xapian.org/ticket/394

I think the key pending issues to resolve in it are to sort out
interactions with other matcher features - this patch should affect only
the search time, not the results, and I think with some combinations of
options it might affect results currently.

We should get that patch applied for 1.3.0 which will give us a
development series to shake out any issues, so I've updated its
milestone.

Cheers,
    Olly