[Xapian-discuss] binary db compatibility between versions

[Olly Betts]

On Tue, Mar 31, 2009 at 04:09:25PM +0100, Richard Boulton wrote:
> On Tue, Mar 31, 2009 at 03:50:01PM +0100, Ben Campbell wrote:
> > Is there a particular policy about if/when the on-disk format for xapian 
> > indexes changes between versions?
> > 
> > (in specific, I'm looking at generating an index offline on a machine 
> > using xapian 1.0.5, then upload the files for use on a machine with 
> > xapian 1.0.7)
> 
> Hmm - I thought we'd documented this somewhere, but I can't find it right
> now.
> 
> The general idea is that on-disk database formats for a stable backend
> (such as "flint" during the 1.0.x release series) shouldn't change between
> releases.  However, we weren't able to maintain that during the 1.0 release
> series due to needing to fix various bugs.

0.9 flint and 1.0 flint aren't compatible due to a bug fix.

But the changes in 1.0.2 were to add support for spelling and synonyms
(and while we were at it, we made the value and position tables lazily
created).  And 1.0.3 added support for user metadata.  So it was new
features not bugs which drove these format changes.

As Richard noted, 1.0.3 can read databases created by 1.0.0-1.0.2 (but
databases created or updated by 1.0.3 can't be read by earlier
versions).  Similarly for 1.0.2 and 1.0.0-1.0.1.

In hindsight doing this was probably a mistake.  We'd assumed that this
would work well for users, as they'd upgrade and not try to downgrade
again.  But it created a lot of headaches for the OLPC folk.  There a
database might be created by a laptop with a newer Xapian and then fail
to be read by another laptop with an older Xapian.

So I think we'll be less likely to do this again, but I'm interested to
hear feedback from users about what sort of compatibility promises they
find useful.

> For the particular revisions you mention, I believe there should be no
> compatibility issues; but I recommend reading the entries in NEWS for 1.0.6
> and 1.0.7 carefully.

Um, building with 1.0.5 and reading with 1.0.7 will be fine (unless
we've messed up on backward compatibility, and there are tests in the
testsuite that reading older format flint databases works so this seems
unlikely).

I think the reverse would also be OK in this case.  Note that cases
which aren't compatible will throw DatabaseVersionError, so just trying
it is the simplest way to check.

Cheers,
    Olly