[Xapian-discuss] another DatabaseModifiedError issue

[Olly Betts]

On Wed, May 28, 2008 at 11:33:39AM +0200, Markus Wörle wrote:
> They occur, if my writer (manually) flush()es two databases with  
> little cached changes consecutively while having high load on the  
> searchers.
> 
> I know that DatabaseModifiedErrors get thrown if a database version  
> get incremented at least two times while a search is running.

Well, *can* get thrown.  Sometimes you'll get away with it (if the
reader only looks at parts of the Btrees which survive the update).

> In fact  
> there *are* two very fast db updates in this case, and there *are*  
> queries that might easily exceed the runtime of those updates. But my  
> updates are on different databases.
> 
> It feels to me, like my reader processes mixing up the version  
> information of the different databases or something like that?

That seems extremely unlikely as they are tracked in separate objects -
it's not like we have a central pool of information here.  I struggle
to think of any way this could happen which wouldn't have manifested
in much worse ways.

The only thing I can think is that perhaps DatabaseModifiedError is
been thrown "too eagerly" - i.e. in situations where the database
isn't actually modified.  But I've not heard any other reports.

Could this be due to the automatic flushing which happens (by default
every 10000 changed documents)?  You can avoid that by using
transactions, or by setting environment variable XAPIAN_FLUSH_THRESHOLD
to a very large value.

Otherwise, what's the minimum time between explicit flushes on the
*same* database object?

And what's the maximum search time?

Perhaps add logging around the flush() and reopen()+get_mset() calls to
make sure these times are as you think they are?

Cheers,
    Olly