[Xapian-devel] Bitsize project: Krovetz Stemmer

Olly Betts olly at survex.com
Sun Feb 15 22:34:07 GMT 2015


On Sun, Feb 15, 2015 at 05:05:11PM +0000, James Aylett wrote:
> How you then structure that in your code as you load it from file and
> later use it is entirely up to you. If it’s just a list of words that
> should be treated specially, having a class to represent each word
> feels like overkill — you can probably do it with something like an
> STL container of a base_string of some sort (std::wstring? I haven’t
> done much Unicode in C++ work, so others may want to jump in and
> correct me here).

Where xapian-core cares about the encoding, it deals with UTF-8 encoded
text, which we store as const char * or std::string.  Using std::wstring
would be appropriate if we were handling wide characters, but converting
UTF-8 to and from a wide character string is likely to end up
significantly slower.  The trade-off is that iterating a UTF-8 string is
more complex than a wide character strings - there it's a simple pointer
dereference and increment per Unicode character.

If you use std::string, that is a class and it represents each word,
which as James says might indeed be overkill.  If the stemming
dictionary is potentially very large, you might want to load the file
into a single allocated block of memory and then just use const char *
into that block for the words - that would avoid the overhead of
creating a huge number of std::string objects.

Cheers,
    Olly



More information about the Xapian-devel mailing list