[Xapian-discuss] "Value in posting list too large" error with 1.1.4 (chert and brass, not flint)

[Olly Betts]

On Sun, Mar 07, 2010 at 06:08:36PM -0500, Jason Tackaberry wrote:
> On Sun, 2010-03-07 at 22:57 +0000, Olly Betts wrote:
> > It seems to me this is certain to be a red-herring.  Except for zero bytes
> > the characters in a term just a string of bytes to the backend.
> > 
> > So it's just coincidence that changing the query avoids the problem.
> 
> Can you elaborate on what the error means and what the likely causes
> are?

Yes, but that requires me to look at the code.  I just addressed this point
directly first to avoid you wasting time on it.

It means we tried to unpack a variable length integer, but the encoded value
was larger than the type can hold.  That probably means we're trying to decode
the wrong data, or junk.  This exception could come from many places though.

I assume since the test data is email that it's not something you can make
available?

If not, you'll need to do the debugging leg work.  I'd suggest trying trunk
first.  I don't see a relevant fix, but it would be annoying to spend time
debugging an already fixed problem.  You can use a snapshot tarball if you
don't fancy bootstrapping from SVN - see near the bottom of:

http://xapian.org/bleeding

If you configure xapian-core without optimisation and with debug info for
easier debugging:

./configure CXXFLAGS='-O0 -g'

Build it, and then run your test program under gdb, e.g. for omindex:

gdb --args omindex --db /path/to/db /path/to/html

Then set a breakpoint on where this exception is thrown, and run it:

b backends/chert/chert_postlist.cc:153
(y to confirm)
run

When gdb stops at that line, get a backtrace with "bt".

Cheers,
    Olly