[Xapian-discuss] can a corrupt database be fixed?

[Olly Betts]

On Thu, Jul 24, 2008 at 05:39:29PM +0100, Ronan wrote:
> I have a Xapian database which appears to be corrupt.  I know this is
> a long shot, but I'm wondering if there's any way to repair it.

I realise you're no longer needing an answer, but others may want to
know.

Do you know how it became corrupted BTW?

Corrupt Xapian databases are very rare in my experience, and the only
cases I'm aware of which weren't attributable to either hardware failure
or concurrent writing after defeating the database write locking (quartz
used to leave stale locks behind if the Xapian-using process didn't exit
cleanly so people had to delete such lock files and sometimes did so too
keenly) are down to mishandling the base files in corner cases.

Such cases are likely to be recoverable by removing whichever of baseA
or baseB is wedged, to leave a consistent set of tables (but take a copy
before you start deleting!)  Or you might find a newer release of Xapian
can read them the "corrupt" database fine.

There isn't a "database repair tool" currently.  If only the postlist is
corrupted, then using copydatabase will generate a copy of the database
without looking at the postlist, except to open it.`

> Why do I think it's corrupt?  Let me present some evidence.
> [...]

Running xapian-check is probably the best check.  That checks the
structure of each table, and also performs some cross-checks for
consistency.

> Alternatively, if there is some programmatic approach which would salvage
> most of the data, that would also be good.

This is the main obstacle to writing a "corrupt database fixer" - I
don't know what the appropriate programmatic approach would be as I've
not seen what corrupt databases look like.  And producing a non-corrupt
database mostly full of junk isn't useful.

> In case it's relevant: the version of the Xapian software I'm using
> is 0.9.9.  To be precise: I'm using a fairly old Debian system, and the
> relevant Debian packages seem to be libxapian13, which is at version
> 0.9.9-0stable1, and php4-xapian, which is at version 0.9.9-1stable1.

I'm afraid that's really too old for us to still provide help with.  We
do supply backported packages of newer Xapian versions for Debian and
Ubuntu though - see the download page for details.

Cheers,
    Olly