[Xapian-discuss] Exception: Key too long

[Olly Betts]

On Thu, Sep 28, 2006 at 09:02:25PM +0200, Markus W?rle wrote:
> I found the problem in my code. It was hard to find because this error 
> seems not to occur until a flush on the disk, and the error message 
> contains no reference to the key where it fails. I patched my libxapian 
> more verbose and got it, finally.

If the term has positional information, Xapian should complain early,
but if it doesn't I think it will indeed not get reported until the next
flush happens.

The key isn't included in the exception because it generally contains
binary data, but this isn't terribly helpful.

It occurs to me that if we just ignore the zero byte issue, we can
fairly easily catch oversize terms at the add_term/add_posting
stage (we just need a "maximum term length" method on the database
backend).  Most terms don't contain zero bytes, so that cleans this
up for the vast majority of users.

> It seems that xapian does not detect the error until it flushes on 
> disk, and if it does, and it fails, it seems that it keeps the 
> unflushed erroring content in memory. After adding a new valid 
> document, it notes that its still over the i-should-flush-now 
> threshold, and tries to flush again, and errors again, and so on...

Hmm, that's unfortunate.  Rejecting early will avoid this happening.

Cheers,
    Olly