[Xapian-discuss] Robust "online" xapian backup?

[Olly Betts]

On Thu, Oct 14, 2010 at 05:03:26PM +0200, Jesper Krogh wrote:
> On 2010-10-14 08:18, Olly Betts wrote:
>> Yes, that should work (though it's better to think of it as "commit" than
>> flush, and we've renamed the method now to reflect this).
>>
>> The obvious approach is a second lock which is taken when actually
>> committing, which should be only for a short interval of time.  The backup
>> would hold this lock (for quite a long time) to prevent changes being
>> committed.
>    
> That would be awesome.. would it tough to implement?

It's certainly conceptually simple enough.  I think there may be some
"devil in the detail", but probably not a lot.

I think rather than document the interface as "you need to take out an
fcntl lock on this file" it makes more sense to keep how this works an
implementation detail - that way it can smoothly work cross-platform,
and with future backends.  So instead you'd have something like:

xapian-lock /path/to/* -- backup --backup-option /path/to

Where xapian-locked is a pretty thin wrapper which would take out commit
locks on the specified databases, and then run the specified command in
a child process.

Currently we just lock the first byte of the flintlock file.  So we can
either lock different parts of that file or have a second lock file.  If
we use the same file, we need to communicate with the child process
which holds the lock (which is possible, as we do hold a pipe open to
it).  If we use a separate lock file, we can probably just instantiate
the class which handles the locking a second time, but that means we
need a second child process for each open WritableDatabase, which
doesn't seem ideal.

And if the commit lock can't be taken, the automatic flushing should
probably just flush the pending changes to disk, but not commit them.
That frees up the memory they're using and means indexing doesn't stall.
This part is very easy to do as we already do this during transactions.

If anyone can see any flaws in the above, or has any better ideas, feel
free to chip in.

Cheers,
    Olly