[Xapian-discuss] Multiple simultaneous WritableDatabases?

[Olly Betts]

On Thu, Apr 27, 2006 at 10:21:28PM -0300, Paul Legato wrote:
> Is there any way that one can write to more than one database at the 
> same time in parallel? I'd like to keep a redundant backup copy of my index.

There's no built-in support, but you could just create multiple
WritableDatabase objects and apply the same operations to each.  There's
no reason why you can't call add_document with the same Document object
on more than one WritableDatabase object.

> If I can't write directly to multiple databases simultaneously, I'm 
> thinking of rsync'ing the database directory to multiple backup servers 
> (which could also serve as read-only slave replicators). Are there any 
> caveats involved with copying a live database around while some process 
> may still be potentially writing to it?

If two or more updates get committed during the copy, you may end up
with a broken database (depending which versions of which blocks get
copied).  One update should be OK.

> Is it necessary to pause the 
> writing processes before copying, or to lock the database in some way?

That's safest.  Just opening the database as a WritableDatabase is an
easy way to prevent it being updated (for quartz you'll have to manually
remove the lock file if you want to update a back-up version, since the
lock file will also then get backed up).

If you're using rsync, you could call rsync repeatedly until no updates
happened during an rsync (if you're backing up nightly, then the first
rsync has a lot to do, but the subsequent ones will just be copying over
the blocks changed by any committed changes since the previous rsync.

Another approach is to use a filing system that allows you to take an
instant snapshot of a file, and then backup the snapshot - that should
always get you a valid database.

Cheers,
    Olly