[Xapian-devel] Database Locking

Olly Betts olly at survex.com
Fri Apr 8 13:27:14 BST 2005


Locking of Quartz databases is currently done with a lockfile.  This
works well from the actual locking side, but the downside is that
the lock isn't released if a process doesn't destroy the
Xapian::WritableDatabase object.  This is made worse because some
of the bindings don't call destructors (or don't do it reliably).

The obvious alternative is to use actual locking APIs.  On Unix, this
mean fcntl.  Windows has a suitable API too.

But the problem is that on Unix, fcntl locks are per process.  So if
we used fcntl locking, then if a process tried to open the same database
twice as a WritableDatabase, it would succeed.  This is probably more of
a worry in a threaded application, but threads aren't actually needed to
hit this problem.

So that's why we've stuck with lock files so far.  But I have a cunning
plan...

We fork() and then fcntl() lock from a different process.  Since each
WritableDatabase now has its own locking process, fcntl() locking works!

Now we don't really want to carry round the VM weight of the whole
application in each lock process, as we'll end up eating up swap with
old copies of pages from the application if there are many long lived
WritableDatabase objects.  So fork() and exec() makes sense.

I wondered about fork() fcntl() exec("/bin/true") and then just kill the
child process to release the lock.  But we want to be sure that the
child process dies when the parent does, so it might be better to have
our own little helper program.  We could open a pipe to the helper,
which then blocks on read or select until the other end of the pipe is
closed either explicitly or by the main process terminating.  Hmm, that
sounds a lot like "/bin/cat" actually!

We could perhaps even allow people to avoid the overhead of fork() and
exec() by allowing a hint to be set.  So if you tell Xapian that you're
not opening the same database more than once for writing, it'll just
lock in process.  It's worth checking if the overhead is actually enough
to be an issue first though.

And on Windows we can just use the locking API.

Cheers,
    Olly




More information about the Xapian-devel mailing list