[Xapian-devel] GSOC 2011- CJK Support

[Olly Betts]

On Thu, Apr 07, 2011 at 11:11:13PM +0800, yong zhang wrote:
> I want to work on this project "CJK Support", I come from Beijing of china.

Thanks for your interest.

In case it isn't clear, you need to submit an application here before
19:00 UTC on April 8th (which is less than 15 hours away now):

http://www.google-melange.com

Just sending a mail to this list isn't enough - you need to be in
Google's system.

> As Chinese letters are not delimited by whitespace, we cannotdistinguish
> them easily. After my investigation, I find three methods to resolve this
> issue, and I prefer the last one.
> 
>    1.
> 
>    Set each letter as a key to index, This is used by Lucene as default.
> 
>    The class is *StandardAnalyzer*
>    2.
> 
>    Every two letter as a key to index. This is used by Lucene for ?CJK
>    support?
> 
>    The java class name is
> CJKAnalyzer<http://svn.services.openoffice.org/opengrok/s?defs=CJKAnalyzer&project=/DEV300_m103>
>    3.
> 
>    Follow the dictionary rule to distinguish group of characters.

The third method should give the best results, but is harder to
implement, and I think will need separate versions for Chinese, Japanese,
and Korean.  Handling words not in the dictionary list is an issue too.

Certainly we're happy to consider proposals for different approaches.
Just explain why you're taking the approach you are.

Cheers,
    Olly