[Xapian-discuss] Clarification of values, data, fields, and prefixed terms

[James Aylett]

On Mon, Sep 03, 2007 at 12:59:55AM +0100, Olly Betts wrote:

> > Values are used for filtering in the match process. So collapsing can
> > be done on a value; you can use them in a MatchDecider and so
> > on. Range filtering is another example, as you point out.
> 
> Sorting results is another big use.

Good point. Must wake up more often.

> > I'm not certain that it is actually true right now, but in theory
> > you'll get better performance in some cases by using values as they're
> > intended (to be looked up and used during the match process), and data
> > as it's intended (to store additional metadata that Xapian doesn't
> > care about, for display/whatever in your application).
> 
> Not just in theory.  Currently to read a value for a document, all the
> other values for that document have to be read, so abusing values as
> general purpose fields will mean that more data has to be read for each
> value accessed during the match - that's clearly going to adversely
> performance in most cases.

I was pretty sure it would be the case, but I didn't want to stick my
neck out without being sure ;)

> I'd like to change how values are stored, but it'll still be a bad idea
> to misuse them - just for different reasons.

The general point is that Xapian core will be optimised towards the
expected use of data, values and terms, so it's better to use them as
designed. Which hopefully is a little clearer now. :)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org