[Xapian-discuss] Query documentation

[Olly Betts]

On Tue, Feb 22, 2005 at 01:50:22PM -0500, Sig Lange wrote:
> I'm looking for a basic example of how to implement a search based on
> specific documents. I am attempting to index a quite large music
> collection. I am interested in search specific sections of this data.
> For example, if I want to search only artist, but also have the
> ability to search everything, Is there a way to perform a query like
> "artist:perfect cicle" that'd only search the indexed artist field?

I've put the attached document together from a few snippets I found and
wrote the rest from scratch.

For your particular case, you might use the prefix XART for "artist".

Cheers,
    Olly
-------------- next part --------------
Xapian itself doesn't put any restrictions on the contents of a term, other
than that terms can't be empty, and there's an upper limit on the length
(which is backend dependent - quartz allows just over 240 characters, except
that zero bytes count double in this length).

However, Omega and Xapian::QueryParser impose some rules to aid
interoperability and make it easier to write code that doesn't require
excessive configuring.  It's probably wise to follow these rules unless
you have a good reason not to.  Right now you might not intend to use Omega
or the QueryParser, not to combine a search with another database.  But if
you later find you do, it'll be much easier if you're using compatible
rules!

The basic idea is that terms won't begin with a capital letter (since they're
usually lower-cased and stemmed) - unstemmed terms which might are handled
by adding an "R" prefix.  So any term which starts with a capital letter is
assumed to have a prefix.  For all letters apart from X, this is a single
character prefix and these have predefined standard meanings (or are reserved
for standard meanings but currently unallocated).

X starts a multi-capital letter user-defined prefix.  If you want a prefix for
something without a standard prefix, you create your own starting with an X
(e.g. XSHOESIZE).  The prefix ends with the first non-capital.  If the term
you're prefixing starts with a capital, add a ":" between prefix and term to
resolve ambiguity about where the prefix ends and the term begins.

Here's the current allocation list:

A	Author
D	Date (numeric format: YYYYMMDD or "latest" - e.g. D20050224 or Dlatest)
G	newsGroup (or similar entity - e.g. a web forum name)
H	Hostname
K	Keyword
L	ISO Language code
M	month (numeric format: YYYYMM)
N	ISO couNtry code (or domaiN name)
P	Pathname
Q	uniQue id
R	Raw (i.e. unstemmed) term
S	Subject (or title)
T	mimeType
U	full URL of indexed document - if the resulting term would be > 240
	characters, a hashing scheme is used to prevent overflowing
	the Xapian term length limit (see omindex for how to do this).
W	"weak" (approximately 10 day intervals, taken as YYYYMMD from
	the D term, and changing the last digit to a '2' if it's a '3')
X	longer prefix for user-defined use
Y	year (four digits)

Reserved but currently unallocated: BCEFIJOVZ

There are two main uses for prefixes - boolean filters and probabilistic
(i.e. free text) fields.

Boolean Filters
===============

If the documents being indexed represent people, you might have a gender
field (e.g. M for Male, F for Female, X for Unknown).  Gender doesn't have
a standard prefix, so you might allocated "XGENDER".  And then lowercase
the field contents to avoid needing to always add a colon.  So documents
will be indexed by one of XGENDERm, XGENDERf, or XGENDERx, and you can
restrict a search in Omega by passing a B parameter with one of these as
the value.

Probabilistic Fields
====================

Say you want to index the title of the document such that the user can
search within the title by specifying title:report (for example) in their
query.

Title has standard prefix S, so you'd generate terms as normal, but then
add an "S" prefix.  You then need to tell Xapian::QueryParser that "title:"
maps to an "S" prefix.  In 0.8.5, you do this like so:

    Xapian::QueryParser qp;
    qp.prefixes.insert(make_pair<"subject", "S">);
    // And similar lines for other probabilistic prefixes...
    // And any other QueryParser configuration (e.g. stemmer, stopper).
    Xapian::Query query = qp.parse_query(user_query_string);

CVS HEAD (and so the next release) change this by abstracting the prefixes map
so you use an add_prefix() method instead:

    Xapian::QueryParser qp;
    qp.add_prefix("subject", "S");
    // And similar lines for other probabilistic prefixes...
    // And any other QueryParser configuration (e.g. stemmer, stopper).
    Xapian::Query query = qp.parse_query(user_query_string);

You can add multiple aliases for a prefix (e.g. title and subject for S), and
the decoupling of "UI prefix" and "term prefix" means you can easily translate
the "UI prefixes" if you have frontends in different languages.

Note that if you want words from the subject to be found without a prefix, you
need to generate unprefixed terms as well as the prefixed ones.