[Xapian-discuss] Robust "online" xapian backup?

[Jesper Krogh]

On 2010-10-14 08:18, Olly Betts wrote:
> On Thu, Oct 14, 2010 at 06:51:28AM +0200, Jesper Krogh wrote:
>    
>> I'm struggling a bit with getting the Xapian indexes safely stored on tape.
>> LVM snapshots is not really an option given the performance penalties
>> of that technology. And the index often encounter writes in the
>> time it takes for the backup to copy off the files. Thus the
>> index is potentially unsafe on tape.
>>      
> There's some discussion here:
>
> http://xapian.org/docs/admin_notes.html#backup-strategies
>
>    
>> I dont know much about Xapian's internal versioning of data, but
>> I suspect it is something like PostgreSQL's MVCC system. Would
>> so my theory was: If no ->flush "has ended" while the backup was
>> running, then the index would be safe, since the last thing to do
>> is more or less to set the revision flag to the new version, which
>> all have to co-exists with a fully functional version?
>>
>> Could we make a hook in Xapian so the backup could signal
>> to the indexer that it is allowed to ->flush the index, but
>> should hold of the "revision flag" until the backup signals that
>> it is done..  Wouldn't that ensure a fully working copy all the time?
>>      
> Yes, that should work (though it's better to think of it as "commit" than
> flush, and we've renamed the method now to reflect this).
>
> The obvious approach is a second lock which is taken when actually
> committing, which should be only for a short interval of time.  The backup
> would hold this lock (for quite a long time) to prevent changes being
> committed.
>    
That would be awesome.. would it tough to implement?
> More crudely, you could just open the database for writing while making
> the backup, which would completely prevent other writers.
>    
Thats not that good.. I have 4 indices to have 4 concurrent writes.
And I really wouldn't like to block the updateprocess completely
while copying data..  neither would I like to hold back the backup
system until the active writers of the current indices have finished
the current run.

-- 
Jesper