[Xapian-tickets] [Xapian] #284: occasional DatabaseModifiedErrors

[Xapian]

#284: occasional DatabaseModifiedErrors
---------------------------+------------------------------------------------
 Reporter:  mrks           |        Owner:  olly    
     Type:  defect         |       Status:  assigned
 Priority:  normal         |    Milestone:  1.1.1   
Component:  Backend-Flint  |      Version:  1.0.7   
 Severity:  normal         |   Resolution:          
 Keywords:                 |    Blockedby:          
 Platform:  Linux          |     Blocking:          
---------------------------+------------------------------------------------

Comment(by mrks):

 I recently refactured our indexing/searching topology to support
 distribution as well as replication, but unfortunately, after that, our
 sleep(4)-workaround did not work reliably anymore. I decided to use the
 new possibilities to replicate our profile-index to another machine with
 completely different (disk/raid-) hardware, but the issue occured again. I
 spend a few days to dig again, and finally I managed to reproduce the bug
 with two little perlscripts that are indepentend from our software or
 data!

 These scripts are:
 http://5nord.org/~mrks/xapian/writer.pl and
 http://5nord.org/~mrks/xapian/reader.pl

 Steps to reproduce:

 1) first run writer.pl - it will build and populate an index in
 /tmp/test_index with 500.000 documents, each with one random 10-character-
 value ([A-Z]) for sorting, and two random terms XFOO... and XBAR... .
 Beyond, writer.pl manually flush()es every 50.000 documents. Wait until
 writer.pl finished this task because the bug will more likely occur while
 documents are replaced in an existing index.

 2) run reader.pl - it will search the index by (endlessly) querying
 "XFOOone AND XBARtwo" and sorting by value_then_relevance on value 0. I
 suggest to run at least three instances at the same time to improve the
 chance of the bug to occur. It will also occur with one single instance,
 but you might have to wait a longer period ot time.

 3) at the same time, run writer.pl, which rebuilds the index by replacing
 documents

 The reader.pl processes will occasionally terminate with "terminate called
 after throwing an instance of 'Xapian::DatabaseModifiedError'" and
 SIGABORTs, while writer.pl is running. Beyond I recognized a increased
 chance to ABORT at the time when writer.pl finishes and exits.

 This is reproducible to me on different machines with different
 architectures, e.g. it occurs on our live-systems (e.g. IBM x3650 with
 Ubuntu Hardy) as well as on my MacBook Pro, with MacOS 10.5.6.

 I do not know if these scripts are exactly minimal to reproduce the bug,
 and I have actually no idea whats happening there, but I assume it might
 have something to do with the sorting by value, because if you leave that,
 it won't occur.

-- 
Ticket URL: <http://trac.xapian.org/ticket/284#comment:11>
Xapian <http://xapian.org/>
Xapian