[Xapian-discuss] Atomic DB rebuilds

[Olly Betts]

On Mon, Oct 04, 2004 at 05:29:21PM +0100, James Aylett wrote:
> On Mon, Oct 04, 2004 at 05:26:34PM +0100, Olly Betts wrote:
> 
> > > But if I use symlinks (and I do) in /var/lib/omega/data to point to the 
> > > db - well as we know, its impossible to atomically update a symlink, so 
> > > I can't update another DB and then symlink to it once done.
> > 
> > Do we know that?
> > 
> > You can certainly create a new symlink with a temporary name, then use
> > rename() to replace the old symlink with the old one.  And according to
> > "man 2 rename":
> > 
> >     If newpath already exists it will be atomically replaced (subject to a
> >     few conditions - see ERRORS below), so that there is no point at which
> >     another process attempting to access newpath will find it missing.
> > 
> > Nothing in the ERRORS section seems to say that symlinks aren't replaced
> > atomically.
> 
> I wouldn't rely on it.

The documentation says it works.  I've used it in the past in critical
situations, and never seen problems.  There's a perl module built on the
assumption it works (File::Transaction::Atomic) and I can't find
anything in Google to suggest it won't work.

> I have a horrible feeling that it won't always work under NFS ...

The only problem is likely to be that you can't trust the return code
from rename, but that's easy to workaround - just check if the source
exists after the rename - if it does, rename failed.

> Didn't Richard write a psuedo symlink backend that took a single file
> to configure it to point to where the actual data was?

There are stub databases.  They're not actually a backend, just a wrapper
around the database factory functions (perhaps Richard's original version
was a backend, but the current implementation isn't).

> So you could rename the _file_ over the old one, which certainly is
> atomic, and will work on all NFS implementations that include lockd
> (at least, AFAIK).

I can't see a reason why a symlink rename should be any less reliable
than a file rename - the operation is essentially identical: the atomic
part just requires replacing the inode number in a directory entry.

Cheers,
    Olly