[Xapian-discuss] TCL Binding working, but db_lock doesn't get removed?

[Olly Betts]

On Thu, Oct 21, 2004 at 03:20:22PM -0700, Eric Parusel wrote:
> I suppose that even if the db_lock file normally was removed, in the 
> event of a sudden reboot it would be possible the db_lock file would 
> still be present?  Which would lock the db from writes?

Unfortunately, yes.  Similarly if you kill an indexer with Ctrl-C,
unless you take special steps to avoid this.

We could use fcntl for locking on Unix, and native locking calls on w32.
Unfortunately, fcntl doesn't work between threads in the same process,
so you need to combine fcntl with mutexes.  I hammered out some ideas
with James over a few beers recently, so this may change soon.  There's
a plan to eliminate DatabaseModifiedError which needs fcntl locking so
there are other benefits.

> I do suppose that multiple writers (no write lock) is out of the 
> question?  (easier said than done I'm sure :) )

It's hard.  Realistically, it needs someone who wants it enough to
convince us it's actually a good idea, and either implement it, or fund
the work.

It's easier to either feed updates into one serialising indexer process,
or to index separate feeds into separate databases, and either search
over the combination or merge the databases prior to searching.

> >The compression schemes used in the tables assume the docids aren't
> >sparse.  Something like an auto-incrementing uid field from a SQL
> >database should work well.  Something like using document hashes or
> >machine IP addresses as the docids probably wouldn't (although there's
> >also overhead for storing and using a UID term...)
> 
> Ok, it is based off of a pgsql sequence, starting from 1, incrementing.
> Very rarely will there be an unassigned id. (failed db transaction, 
> possibly, I think)

That'll be fine.  Xapian itself keeps a high water mark, and allocates
the docid just above it.  So if you delete documents, there will be
gaps.

Cheers,
    Olly