Formulating Advanced Queries with Xapian-Omega

[Olly Betts]

On Thu, Dec 29, 2016 at 05:44:50PM +0100, Giulio Teslano wrote:
> a. What other types of extended wild card(s) options are there ?
> 
>    or is this still currently limited to these two characters '*?' ?

As I said, the branch "adds support for arbitrary glob-style wildcard
patterns (where * matches 0 or more characters and ? a single                                                                                                                                                                                
character)".

> b. Apart from 0 or more and single char options are there any other
> options ?

Not that are currently implemented on that branch.

> Were you suggesting that one possibility would be trying something
> similar to : 
> 
> isbn:?-???-?????-? as a very loose general query for ISBNs ?
> 
> (so long as the option is enabled).

It seems you must be talking about the query the user would write here,
but then I'm not sure what the "isbn:" prefix would map to.

But yes, that's the sort of pattern you'd have to use.

One wrinkle with this is that (assuming you use the Xapian::TermGenerator
class) "-" is a word separator character at index time - i.e. you'll get
terms from 1-234-56789-0 and OP_WILDCARD only matches within a term.  So
you need different word splitting behaviour for this to work, which
currently means you'll need to do it yourself instead of using
TermGenerator as that isn't currently configurable.

> 1 Could you mention how one enables and can take advantage of your
> extended option in Omega and/or Xapian ? (working example ?)

Currently you need to use one of the WILDCARD_PATTERN_* constants
when constructing an OP_WILDCARD Query object, e.g.:

Xapian::Query wild(Xapian::OP_QUERY_WILDCARD, "?-???-?????-?", 0, Xapian::Query::OP_WILDCARD_GLOB);

There isn't yet any integration into omega (or even into
Xapian::QueryParser).  Such are the pitfalls of using code from unmerged
branches I'm afraid.

> 2 The ? Wild Char is for general characters, is it not ?
> 
>   ie. It cannot distinguish between digits and letters and thus cannot
> act as a RE \d or [0-9] ?

"?" matches any single character.  The project this branch is for only
required allowing "*" anywhere in a term (rather than it only being
supported at the end) and adding support for "?", so there's not
currently a plan to support pattern styles other than globbing, or
additional glob-style patterns.  The flags to control this were picked
such that either could be done in the future.

> > If you have particular "code" patterns which are important in your domain,
> > I'd consider pulling them out at index time and adding them as a filter
> > term
> 
> It is no doubt due to my lack of understanding but how would this
> interesting option 'pulling them out at index time ...' be implemented
> ?

For example in Perl, at index time:

	while ($text =~ /(\b\d-\d{3}-\d{5}-[\dX]\b)/g) {
	    $doc->add_boolean_term("XISBN$1");
	}

With this approach, you could also easily do additional validation (such
as checking the check digit for codes which have one, as ISBNs do).

Then at query time:

        $queryparser->add_boolean_prefix("isbn", "XISBN");

Then the user can use isbn:1-234-56789-0 to filter only documents
mentioning that ISBN.  Or if you want to be able to find documents
which mention any ISBN (or anything which looks like one) then:

	if ($text =~ /(\b\d-\d{3}-\d{5}-[\dX]\b)/) {
	    $doc->add_boolean_term("XHASisbn");
	}

Then at query time:

        $queryparser->add_boolean_prefix("has", "XHAS");

And then the user can filter a search by: has:isbn

> It would be very useful if there were some working examples in
> relation to these themes, (at least for those less expert than the
> xapian developer level). Xapian-Omega appears to be a very interesting
> solution and with an RE option it would be one of the most flexible
> and versatile SEs currently available on the net

I suspect that most end users wanting "regexp search" don't just want to
search for terms matching a specified regexp (which is how OP_WILDCARD
inherently works), but rather to perform regexp matches over the whole
document (like https://codesearch.debian.net/ does for source code).
To do that efficiently you need a different index structure.

Cheers,
    Olly