[Xapian-devel] GSOC 2012 : QueryParser Reimplementation

[Dan Colish]

On 3/25/12 6:42 PM, Olly Betts wrote:
> On Sun, Mar 25, 2012 at 09:27:19PM +0530, Sehaj Singh Kalra wrote:
>> I have a question - Does Xapian at present cache the result at any level ?
>> If not, then I can add the functionality of caching ("after" the Query has
>> been parsed as you rightfully explained that it's probably better to cache
>> after the QueryParser) in my proposal.
> Xapian doesn't explicitly cache results between queries, though the
> operating system will cache recently read disk blocks, which makes a
> big difference - if you turn on the machine and time running the same
> query twice in a row, it'll be much faster the second time, entirely
> due to this.
>
> Make sure you don't bite off too much though - it would be better to
> fully implement a replacement QueryParser and get it merged than to
> partly implement a replacement QueryParser and partly implement a query
> result cache.  The current QueryParser supports quite a lot of features
> and those all need to work at least as well in the replacement as they
> do currently before we can make the switch.
>
Hi Sehaj,

Before the caching idea gets too far down the wrong path, I'd strongly
like to suggest that we don't get into result caching inside the
QueryParser. As Olly pointed out, we already do page caching which is
extremely efficient and caching is typically application specific. I see
no need to do more internally.

Please take into account how much work simply reimplementing the
QueryParser will be and whether you want to reimplement it or add to the
existing implementation. Your idea to add natural language support would
be interesting and potentially achievable for English; there are
libraries like LinkGrammar which provide parsing solutions.

Finally, I'm not sure what use those patents will be. Any involvement
with them will surely be a bad choice due to their legal complications.


--Dan