[Xapian-devel] Database Locking

[Olly Betts]

On Fri, Apr 08, 2005 at 01:27:14PM +0100, Olly Betts wrote:
> So that's why we've stuck with lock files so far.  But I have a cunning
> plan...
> 
> We fork() and then fcntl() lock from a different process.  Since each
> WritableDatabase now has its own locking process, fcntl() locking works!

I've now got round to trying this, and it works pretty much as I'd
hoped.

It needed one slight tweak - we seem to require a bidirectional
connection so that the child process can tell the parent if fcntl()
failed or not.

Without a back connection, the child process can die to say the lock
failed but the parent won't be able to tell if the child has got the
lock or is still trying.

And without a forward connection, the child won't know when the parent
exits without explicitly closing the lock.

There may be a way around this, but a bidirectional connection isn't
really a problem.

> I wondered about fork() fcntl() exec("/bin/true") and then just kill the
> child process to release the lock.  But we want to be sure that the
> child process dies when the parent does, so it might be better to have
> our own little helper program.  We could open a pipe to the helper,
> which then blocks on read or select until the other end of the pipe is
> closed either explicitly or by the main process terminating.  Hmm, that
> sounds a lot like "/bin/cat" actually!

What I'm currently doing is execl("/bin/cat", NULL); and then if that
fails I just enter a loop to read bytes from stdin until EOF.

> We could perhaps even allow people to avoid the overhead of fork() and
> exec() by allowing a hint to be set.  So if you tell Xapian that you're
> not opening the same database more than once for writing, it'll just
> lock in process.  It's worth checking if the overhead is actually enough
> to be an issue first though.

Here are some preliminary timings for a successful lock and unlock on
x86_64 Linux (I timed 10000 iterations using "time"):

 98.4-175.2 us   Quartz style locking with a lock file
533.2-605.6 us   fork/fcntl/exec locking
  8.1-10.4  us   fnctl alone

("us" being microseconds)

Although the fork/fcntl/exec scheme is around 3-6 times slower than
quartz, I suspect it's not actually a problem for WritableDatabase
becase the lock/unlock still takes only half a millisecond.

Now the locking works, I'll see if I can streamline it at all.  And also
get timings on other platforms.

Cheers,
    Olly