[Xapian-discuss] DatabaseModifiedError

[Andrew Betts]

On 23 Mar 2011, at 10:35, Richard Boulton wrote:

> On 23 March 2011 10:16, Andrew Betts <andrew.betts at assanka.net> wrote:
>> Hi,
>> 
>> I am starting to see more frequent occurrences of a DatabaseModifiedError when users execute searches on my site.  Since our index is being updated all the time, its easy to understand why these happen, but is it possible to ignore this error and execute the search against the stale revision?  It's likely only a few seconds out of date, and if I follow the advice and reopen() the database, it's quite possible that it will have been modified again before I manage to call get_mset.
> 
> Xapian can't be told to ignore the error and use the stale revision,
> because the error is only thrown when the data Xapian needs from the
> old revision has already been overwritten by new data.  If you're
> searching on version N, you won't get the error until before version
> N+1 has been committed.  After version N+1 has been committed, changes
> to the database (ie, leading to version N+2) will start overwriting
> blocks which were used in version N (but not in version N+1), and
> there is then a risk of the exception being thrown.
> 
> Assuming you're updating the index every few seconds, and searches
> take no more than a couple of seconds in the worst case you could
> perhaps insert a pause after committing a change before performing any
> further modifications to the database, to allow readers a chance to
> reopen on the new revision before risking overwriting blocks from the
> old revision.

Hmm.  The problem is that, to give us high availability, the process works like this:

1. Editor clicks save on a post.
2. Indexing job is posted to a RabbitMQ message queue
3. Two instances of a search indexer daemon, running on different servers, are consuming the queue.  One of them will pick up the job
4. The daemon opens the Xapian db from a distributed shared filesystem, makes the change and commits it.

So I can't guarantee a pause in indexing of any particular length because there are multiple independent indexers.  Is there a way around this or should I reconsider this architecture?