[Xapian-discuss] Is there a 64 character term size limit? In Ruby bindings?

[Henry C.]

On Tue, June 8, 2010 03:32, Olly Betts wrote:
> On Mon, Jun 07, 2010 at 07:38:08PM +0100, Francis Irving wrote:
>
>> I've just found some items in my Xapian database which aren't being
>> indexed, when the terms are quite long.
>>
>> Example term:
>> Frotherham_doncaster_and_south_humber_mental_health_nhs_foundation_trust

I've run into this as well using the Perl bindings.  I found discussions
regarding this with Omega where truncate/hash is used (correct me if this
not related).

My issue is that exceptions (ie, "Exception: Key too long: length was...")
are sometimes only thrown when indexing is complete and I flush/close.  I
may be wrong about when it's thrown since I'm looking at log files.  Other
times, the exception will occur followed by another: "Unexpected end of
table when reading continuation of tag..." -- this is probably because of
the unhandled initial exception.

Anyway, when using Perl, how can I either truncate (to say, 239) or hash
the key to prevent this error from occurring?  My real-world data can be
quite dirty, so I need to gracefully handle this issue.

Thanks
Henry