[Xapian-devel] multiple writers and remotetcp backends

[Richard Boulton]

Olly Betts wrote:
> On Tue, May 01, 2007 at 12:36:54PM +1000, Mark Hammond wrote:
>> The problem is that a xapian tcp-server in 'writable' mode makes no attempt
>> to ensure only one 'active' connection at a time is trying to modify the
>> database.  If multiple connections are made to a writable server, the
>> behaviour is undefined (or even it is was defined, it is unlikely to be
>> defined in a way that would make it useful).
> 
> I'd not appreciated this happened from the previous discussion - this
> is certainly a bug.  I understood the issue was just that of trying to
> marshal multiple processes wanting to write to the same remote server
> in a sane way.

I thought that was the problem too until looking at the code this morning.

> Looking at the code, I believe it's also wrong that we open the database
> and then fork multiple processes which can make use of it, even for a
> read-only Database.  We certainly don't promise that you can use the same
> Xapian object from different threads.  I think similar rules ought to
> apply over fork.

Yes - I would imagine that the behaviour over fork() is system dependent 
- it works okay for readers on Linux, but may well not on other platforms.

> But this matters much more for writers - with the current backends, it
> happens to work OK for readers I think.

I'm dubious that it will work correctly under load even with just 
readers on Windows, having looked at the code, due to it being a 
threaded rather than a forking model.

> So we should probably leave the reader issue for now, as it can be fixed
> without API or ABI changes, but fix the writer issue.

I imagine that a simple fix would be to open the database when a new 
connection comes in, instead of opening the database when the server is 
started and passing it to each sub-process (or sub-thread on windows). 
That way, the normal locking code will be able to enforce the necessary 
constraints.

This will mean that a server will only allow one connection at a time if 
the writable parameter is set, and should throw an exception to clients 
that try to open a second concurrent connection.

If performance is found to be a problem, we can implement a pooling 
system or similar at a later date.

Thoughts?  Or shall I just try implementing it and check if it works as 
desired?

-- 
Richard