[Xapian-discuss] How to update DB concurrently?

[Olly Betts]

On Wed, May 17, 2006 at 04:03:58PM -0500, Peter Karman wrote:
> Olly Betts scribbled on 5/17/06 2:17 PM:
> 
> >This "spool directory" style of design is both simple and suitably
> >robust.
>
> a good idea. probably need to do some file locking though, lest you feed 
> a .dump file to scriptindex that your spooler is in the middle of 
> writing to.

No need, just build the files in another directory on the same
filesystem and use the rename() function to move them into the spool
directory (it needs to be on the same filesystem for rename() to be
atomic).

Or even build them in the spool directory itself with ".tmp" on appended
and skip those in the indexer (but I tend to favour separate directories
as directories with lots of files are slower, significantly so on so
filing systems.)

The only slight wrinkle is that on NFS rename() might return -1 even
though it succeeded, as the Linux man page explains:

    If the server does the rename operation and then crashes, the
    retransmitted RPC which will be processed when the server is up
    again causes a failure.

But so long as you're aware of this issue, it's easy enough to work
around without incurring extra cost in the normal case.  In perl,
something like:

  while (!rename($tmp, $final)) {
    # Check if the rename really failed, or NFS is just being lame.
    last unless -f $tmp;
    # Handle the error condition by just waiting 5 minutes and retrying.
    print STDERR "*** rename($tmp,$final) failed ($?)\n";
    print STDERR "*** will retry in 300 seconds\n";
    sleep(300);
  }

Cheers,
    Olly