[Xapian-devel] Full MVCC in Brass

[Olly Betts]

On Wed, Aug 20, 2014 at 10:22:57AM +0200, Adam Sjøgren wrote:
> Austin Clements <aclements at csail.mit.edu> writes:
> 
> > Full MVCC would enable Xapian to keep any database revision valid as
> > long as any reader is using it, thus eliminating the dreaded
> > DatabaseModifiedError and simplifying application logic.
> 
> This would also make it easier to spool a consistent backup to tape of
> an index that is being used (updated), right?

If you mean for copying at the filesystem level, not really.

If you had a read lock held for all the time the backup was happening,
you should ensure the backup contained enough information to recreate a
working database from, but there's still no guarantee the backup itself
would be a valid database.

You could produce a dump of the entries from each table using
this feature, and then restore by reinserting those - this would
need two special tools, and a way to get data from them to/from the
backup (saving it to a file seems clumsy, but would work if you have
spare disk space comparable in size to the database).  The backup and
restore operation would be comparable in speed to compaction + the time
to do a file-level backup and restore (it would be very like compaction,
except with the data being saved and restored in the middle).

For a large and actively updated database, keeping the revision around
while it gets backed up could inflate the database size quite a bit.
That space would get reused after the backup is taken though.

Another approach to allowing backups of a live database would probably
be to make use of the replication changesets.  If you just backup the
database at the filesystem level (without worrying about locks) then
also backup any changesets created while the backup was running (and the
next one created afterwards), then you should just be able to replay the
changesets onto the database to restore.

If you're replicating already, this is clearly a nice approach.  Not
sure how it would compare if you weren't.

Cheers,
    Olly