[Xapian-discuss] xapian database corrupt with multi-write

[Olly Betts]

On Tue, Nov 01, 2011 at 10:27:11PM +0800, ????????? wrote:
> I found sometimes  DatabaseLockError raised but the database is OK. And
> sometimes database is corrupt.

This is the first report of such an issue I've heard, and we've used the
same locking approach for a long time, so I think there's something
particular to your setup which is the key to what is going on.  So if
you want us to help you, we need to work out what that is, and in order
to do that, you're going to need to provide precise answers to the
questions we ask...

> If I run my app by mistake, it is very often that the database will be
> corrupt. And I am very confused that why the lock file is opened by 'cat'.

The lock is held by a "cat" process because fcntl locking has a
design-flaw which means we need to hold the lock in a subprocess to
avoid it getting accidentally released, and exec-ing cat avoids dragging
round pointless VM overhead.  It's really just an implementation detail
you shouldn't need to worry about.

> > Indeed - what platform (operating system and filesystem) is this on?
> >
> Ubuntu 8.04.4

*and filesystem*?

> > Also, what Xapian version?
> 1.27

There hasn't been a version 1.27.  Since we didn't get to 1.0.27, so I
guess you mean 1.2.7?

> > > Are you getting the DatabaseLockError raised, and then on subsequent
> > > open finding the database to be corrupt? Or are you not seeing the
> > > exception and subsequently finding the database to be corrupt?
> >
> > Yes, a precise explanation of what you see would be very helpful too.
>
> Sometimes I killed our server, but I found the file is still locked by
> 'cat'. It is very strange.
> Maybe the database is corrupt after I start the server again.

Um, that doesn't answer what James asked.

Are you saying that when the second writer tries to get the lock it
gets DatabaseLockError?  Or that it manages to get the lock despite
there already being a writer which should have the lock?

If you're able, showing us the *COMPLETE* code to your indexer would
probably be helpful too.  Just pasting snippets of code isn't much help,
as we can't try the code then, and you may omit something relevant to
the problem.

Cheers,
    Olly