[Xapian-devel] multiple writers and remotetcp backends

[Olly Betts]

On Tue, May 01, 2007 at 01:42:08PM +0100, Richard Boulton wrote:
> Olly Betts wrote:
> >Looking at the code, I believe it's also wrong that we open the database
> >and then fork multiple processes which can make use of it, even for a
> >read-only Database.  We certainly don't promise that you can use the same
> >Xapian object from different threads.  I think similar rules ought to
> >apply over fork.
> 
> Yes - I would imagine that the behaviour over fork() is system dependent 
> - it works okay for readers on Linux, but may well not on other platforms.

I believe it should be OK currently.  But (for example) it will be a
problem when we start trying to lock the revisions which readers are
currently using, so we want to avoid people relying on it working.

> >But this matters much more for writers - with the current backends, it
> >happens to work OK for readers I think.
> 
> I'm dubious that it will work correctly under load even with just 
> readers on Windows, having looked at the code, due to it being a 
> threaded rather than a forking model.

Yes, that could well be a problem already.

> >So we should probably leave the reader issue for now, as it can be fixed
> >without API or ABI changes, but fix the writer issue.
> 
> I imagine that a simple fix would be to open the database when a new 
> connection comes in, instead of opening the database when the server is 
> started and passing it to each sub-process (or sub-thread on windows). 
> That way, the normal locking code will be able to enforce the necessary 
> constraints.

Yes.

> This will mean that a server will only allow one connection at a time if 
> the writable parameter is set, and should throw an exception to clients 
> that try to open a second concurrent connection.

Yes.  Ultimately I think we want a writable reader to be able to specify
if the server should fail right away or block if the database is locked
but failing is certainly an improvement over what is currently happening
and will do for now.

> If performance is found to be a problem, we can implement a pooling 
> system or similar at a later date.

If opening a database is "too slow", we should first try to address that
head on as that would benefit non-remote users too.

> Thoughts?  Or shall I just try implementing it and check if it works as 
> desired?

Go for it.

Cheers,
    Olly