[Xapian-devel] Full MVCC in Brass

[Olly Betts]

On Wed, Aug 20, 2014 at 10:40:17PM -0400, Austin Clements wrote:
> I haven't seen anything saying that LockFileEx interacts with CreateFile
> locking, but if this goes in a new BrassLock, then it shouldn't be an
> issue because we'll always use LockFileEx for brass databases.

If it's feasible, keeping the lockfile called "flintlock" and the
locking interoperable is helpful - it avoids issues like ending up with
two databases of different formats in the same directory, as I've seen
happen with flint and quartz (quartz used a "db_lock" as the lockfile
name and the presence of the file indicated the lock was held).

However the new locking code can certainly be a different class if that
works best.

I doubt LockFileEx() directly interacts with CreateFile(), but you
need to open the file before you call LockFileEx().  And if current
Xapian has the database locked on Windows, then the lock file can't be
opened for write (because only FILE_SHARE_READ was specified).  So when
future Xapian tries to open to write, it won't be able to (it won't even
get to the LockFileEx() stage).

And I think we can continue to have writers open with only
FILE_SHARE_READ to get locking to work the other way around (readers
would I think need to open with FILE_SHARE_READ|FILE_SHARE_WRITE).

> Okay.  If we're on a platform that can't support MVCC, should an attempt
> to open the database for read *without* opting out fail?  Alternatively,
> it could fall back to old-fashioned read locking (blocking writers) so
> that readers still get the guarantee that DatabaseModifiedError will
> never happen (and they could still opt out of all read locking and deal
> with the consequences).

Actually, while I said "opt out", there is the question of what the
default should be (MVCC or DatabaseModifiedError).  MVCC seems more
appealing, though I'm not sure how the much issues with long-lived
readers bloating database size will catch people out.

But if MVCC is the default, I'm not sure what's best to do on a
platform which can't support it.  Failing seems the safest option,
at least for now, but readers blocking writers isn't so bad - better
than readers getting DatabaseModifiedError on some platforms but not
others.

> I think for a first shot I just won't do lock extension.  The next step
> would be to do it unless we're using F_SETLK.  Then the last step would
> be to add a lock helper.  Steps 2 and 3 are backwards-compatible, so
> they could be done at any point in the future.

OK.  I'm generally a fan of an incremental approach.

Cheers,
    Olly