[Xapian-discuss] Error msg xapian-compact: The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation

Olly Betts olly at survex.com
Sat Jun 10 02:07:45 BST 2006


On Fri, Jun 09, 2006 at 10:52:09AM -0800, oscaruser at programmer.net wrote:
> The disk has write cache enabled. I will disable this.

It's a bit of an outside shot, but perhaps worth a try.

> I've noticed from strace that when I have many scriptindex processes
> running concurrently on different flint dbs the fsync operation takes
> about 30 seconds to a minute to complete at times. This has had a
> noticable impact on my rate of processing.

This probably just means that it takes 30 seconds for the disk subsystem
to flush all the cached blocks from the writeback buffers, which tends
to suggest that fsync and write caching should work for you.

Indexing speed will generally be I/O limited, so this sort of thing is
to be expected.

> I was running 150 spiders in
> parallel, but had to scale back to 30 or less. My hope is with this
> disabled the updates will be occur at a faster rate, but it could be
> disabling the writeback cache would not really help that.

Disabling the write cache should largely eliminate the slow fsync, but
only by slowing down all writes.  I find it hard to see how it would be
faster overall without it.

Have you tried setting XAPIAN_FLUSH_THRESHOLD in the environment (and
exporting it)?  It defaults to 10000, but it you've plenty of memory
you can go to 100000 or probably more.  This increase the interval
between automatic flushes.

> Can I modify the copy database tool to ignore the revision check in
> order to attempt to recover the flint index?

The inconsistent revision is detected deep in the flint backend.
Just grep for the error message to find where it comes from...

You could try to just ignore it, but I'm not optimistic that this would
work - the block which has overwritten the one you want is likely to be
from a completely unrelated part of the B-tree.

Did you look to see if you have a full set of both .baseA and .baseB
files?  If so, my suggestion from a previous mail is most likely to
save you.

Cheers,
    Olly



More information about the Xapian-discuss mailing list