[Xapian-devel] buffered tables, sessions, and transactions

[Olly Betts]

Quartz has a QuartzDiskTable class which is a thin wrapper for a pair of
Btree objects (or just one if the table is opened readonly):

http://www.xapian.org/docs/sourcedoc/html/classQuartzDiskTable.html

There's also a QuartzBufferedTable class which adds memory buffering of
changes to this:

http://www.xapian.org/docs/sourcedoc/html/classQuartzBufferedTable.html

However, as of 0.8.0 we now buffer changes to the posting lists in
QuartzWritableDatabase (in the "Private attributes" totlen_added through
to mod_plists):

http://www.xapian.org/docs/sourcedoc/html/classQuartzWritableDatabase.html

This probably removes the main advantage of having QuartzBufferedTable.
If we're adding new documents to a database, it probably doesn't help us
at all.  It probably still helps efficiency a little for scattered
updates to a database (as these will be applied sorted by key, which
is probably a small win).

However, all these changes are being buffered in memory.  We could use
that memory to cache more posting list changes, or just let the OS use
it to cache more disk blocks.

So I propose stripping out QuartzBufferedTable.  I believe any remaining
benefit it provides is small, that it uses a lot of memory which could
be better used, and that it's really just unneeded code which serves to
make quartz harder to understand and debug.

In place of this, changes to all but the posting list would be written
straight to the btrees on disk, but wouldn't be visible to readers until
the changes were flushed.

Opinions?

While looking at this, I noticed the currently unimplemented transaction
methods.  The idea is to allow a group of operations to be specified as
being applied as a unit.  Either they all are applied, or none are.

http://www.xapian.org/docs/apidoc/html/classXapian_1_1WritableDatabase.html#a7

This is actually very easy to implement like so:

* begin_transaction flushes any pending changes and sets an
  "in_transaction" flag

* if "in_transaction" is set, we ignore explicit flushes and don't autoflush

* cancel_transaction closes and reopens the Btree table (or calls a
  new method of Btree which has this effect but without all the file
  descriptor and memory allocation releasing and reobtaining)

* commit_transaction flushes any pending changes and clears the 
  "in_transaction" flag

That's a lot of flushing though - the currently specified API makes
transaction inherently inefficient I think.

If we *just* want to provide a way to specify an atomic grouping, we
could modify the API to allow such transactions with a minimal overhead
like so:

* Remove cancel_transaction

* Rename commit_transaction to end_transaction

* flush is deemed as invalid (or ignored) when in a transaction

* begin_transaction sets an "in_transaction" flag

* if "in_transaction" is set, we don't autoflush

* (Possibly) if we're close to autoflushing already, begin_transaction
  could autoflush

The only loss of effiency now is a tiny housekeeping overhead, and the
loss of autoflushing during the transaction (which is only going to be
noticable for a large transaction).

Thoughts?

And lastly, begin_transaction's docs currently say:

  "A transaction may only be begun within a session, see begin_session()."

This method isn't on the public API.  Sessions are mentioned elsewhere
in the public API docs, but I don't see them as relevant without methods
to control them!  I propose to remove these references (and perhaps also
the session mechanism in the backends).  Alternatively we could add
sessions to the public API.  This would potentially allow multiple
concurrent writers to the same DB with locking to prevent collisions.
But I'm not sure that's appropriate for Xapian.  We're not trying to
build a relational database here...

Again, what do you all think?

Cheers,
    Olly