[Xapian-discuss] xapian-tcpsrv php in real fastcgi loop

[Martin Regner]

Am Montag, 22. März 2010 02:40:06 schrieb Olly Betts:
> On Sun, Mar 21, 2010 at 11:57:03AM +0100, Martin Regner wrote:
> > Now i randomly get an DatabaseLockError when one of the loops is trying
> > to index something (on the writing instance). I try to solve this by
> > implementing a few retries after some microseconds. But this is like
> > betting on a lucky chance. Is this a problem inside the tcpsrv or is it
> > (like before) the not yet realy closed client connection of the
> > loop/thread?
> 
> It's hard to be sure just from the description, but I suspect it is the
> delay between the client closing the connection and the server noticing and
> closing the WritableDatabase object, releasing the write lock.
Yep, it seems so. I deployed the retry solution on Sunday and had no locks 
anymore since then.
> 
> In 1.1.x, you could explicitly call close() on the remote database in the
> client.  I'm not sure I see a better fix than retrying for 1.0.x.  An
> explicit call to flush() before closing will help, and is good practice as
> it means you actually see any errors in committing the batched changes.
I implemented an ebuild for 1.1.4 on our devserver and it runs flawlessly. The 
RPMS for our realy old liveserver won't build yet.

> 
> > Doeas anyone have another approach for multiple clients trying to write
> > simultanously?
> 
> Serialising all writes through a single writer is the usual approach to
> this. If you put updates into some sort of queueing system, you also get
> the ability to shut off the application of updates temporarily which can
> be handy.
> 
> > May it's possible that xapian-tcpsrv is buffering the write requests
> > until the lock is removed (i know this is much more complicated than it
> > sounds).
What i mean is, what you have written above. The idea was a queueing/buffering 
solution inside tcpsrv. The tcpsrv daemon retrieves an 'index-this-document' 
command, takes the data and sends it to the database if write lock on the 
database is released. If there is no lock it behaves as before. So what i mean 
is that the tcpsrv is handling the queueing. It's just an idea without knowing 
what exactly the tcpsrv daemon is doing. 

> 
> I'm not quite sure I follow.  If you're suggesting an explanation of the
> issue, then the OS will buffer writes and then calls to fdatasync() will
> mean we wait for them to hit disk before the lock gets released.
> 
> If you're suggesting a solution, I don't understand exactly what you are
> proposing.
> 
> Cheers,
>     Olly