Commit error with Xapian 1.4.11

[Olly Betts]

On Mon, Aug 26, 2019 at 09:42:09AM +0200, Jean-Francois Dockes wrote:
> A Recoll user gets the following message while indexing:
> 
> "Attempted to delete or modify an entry in a non-existent posting list for #bannerholder"
> 
> The exception happens during a commit call. Xapian version 1.4.11, Debian Buster
> 
> A little more detail here: https://opensourceprojects.eu/p/recoll1/tickets/108/
> 
> I asked if this was reproducible, and to run the indexing in single-thread
> mode to simplify the situation.

It's worth running xapian-check on the database to see what it reports.

Also might be interesting to check what the posting list for that term
is:

xapian-delve ~/.recoll/xapiandb -vv -t '#bannerholder'

> I'm not too sure if a Recoll bug could cause this, or if this has to be a
> Xapian issue, I can open a ticket if more appropriate.

It shouldn't be possible to cause this via valid use of the API, but
bugs in the application could - for example a stray memory write, or
writing to a file descriptor which is open for writing by Xapian (most
likely case is probably a stale handle which the application has closed
and was reallocated to Xapian).  Since 1.3.4 we avoid fds < 3 for
writable database tables, which at least means writing to a closed
stdout or stderr can't corrupt the database now, but it's hard to fully
protect against writes to our fds (https://trac.xapian.org/ticket/651
has some ideas).

There aren't any currently known database backend bugs present in
1.4.11, but it certainly could be an unknown Xapian bug - if it's
reproducible with non-private data I'm happy to take a look.  

If the database was created with an older Xapian version originally, it
might be due to an already fixed bug (e.g. the cursor handling one fixed
in 1.4.7) and the corruption has just escaped notice because nothing
tried to read or update that part of the database since).

We've also seen database corruption reports which turned out to be
caused by hardware issues.

Cheers,
    Olly