Pull requests: CJK words and Snippet generator

[rsto at paranoia.at]

Hi James,

thanks for the feedback.

On Thu, Jul 28, 2016, at 00:22, James Aylett wrote:
> This sounds great! I know sufficiently little about CJK that I won't
> try to comment on that at all :)

I've just opened a pull request for the CJK tokenizer:
https://github.com/xapian/xapian/pull/114
 
> I wonder if we can arrange suitable defaults to use your
> implementation with the older API, and come up with a newer API that
> allows a SnippetGenerator class to be used from the MSet.

The FastMail snippet generator has been written when MSet didn't create
snippets. I'll first compare both implementations to see if there is a
good reason for them to coexist, or might just as well merge any
additional features into MSet.

> A good start would certainly be rebasing against master and opening a
> pull request for each on github (this will trigger travis CI builds,
> which is a helpful first pass in making sure everything good; it runs
> against both G++ and Clang, which can expose some weirdnesses).

Unfortunately, Travis breaks since pkg-config can't find libicu on the
machine [1]. I could make the libicu dependency optional, and that might
be useful for Xapian installation that don't bother with CJK text, but
for Travis tests it would make sense to enable ICU.

Cheers,
Robert

[1] https://travis-ci.org/xapian/xapian/jobs/148268282#L1522