Remote protocol abstracted

[Germán M. Bravo]

I wanted to ask about integrating the changes I worked on (regarding
exposing the remote protocol) and also about replication (which still needs
some work, but it's already working).

The remote protocol and replication parts are actually needed so indexation
clusters work. Please, if you have any thoughts or ideas about how to get
this ball rolling as soon as possible, let me know. I'm very interested in
getting those in before the next major release, so I'd love to hear if you
have any suggestions or anything so I can work them out.


I also want/need to boost the overall performance in Xapiand, by having a
way of making all readers in the same process (different threads) read the
same things the writer reads (even before commit)... I do have some some
ideas about it (using some sort of "subscription" scheme)... but this is a
topic for other thread.


On Wed, Apr 1, 2015 at 5:03 PM, Germán M. Bravo <german.mb at gmail.com> wrote:

> I have been working in the abstraction of the remote protocol to separate
> it from the server and the connection.
> The purpose of this is I need to use the Xapian remote protocol in a
> server we are working on (named Xapiand:
> https://github.com/Kronuz/Xapiand) which is similar to some extent to the
> `xapian-tcpsrv` in that it also serves
> databases remotely through TCP, using the binary protocol. What is
> different about it is that it can serve any
> number of databases using an event driven mechanism for receiving and
> attending clients. Additionally, it also
> serves clients using a custom HTTP protocol which is JSON based and
> RESTful.
>
> I have the following commits in my Xapian fork at
> https://github.com/Kronuz/xapian/commits/master:
>
>
> 1. Remote protocol with support for selection of database directory
> -------------------------------------------------------------------
>
> If the user requests to open a remote database server, it can optionally
> require
> the remote backend to tell the server what database it wants to open, if
> the
> remote server is configured as a server without a default directory
> (xapian-tcpsrv remains to always have a default directory at the moment).
>
> The binary protocol was changed in the way that a new command can be sent
> to
> the server (MSG_SELECT) in case the server tells the client no default
> directory
> is selected, the server then reacts to this command by opening and
> selecting the
> requested database as the active database.
>
>
> 2. Split msg_query and msg_getmset
> ----------------------------------
>
> Originally, `msg_query()` handled two commands by the client: `MSG_QUERY`
> and
> also `MSG_GETMSET` (which was marked as "used during a conversation"). This
> changes this and makes it so `msg_query()` returns even when the client
> hasn't
> yet sent a `MSG_GETMSET`, but instead only keeping the current satate of
> the
> query in the `MatchState` struct and setting the expected `required_type`
> for
> the next command.
>
> The purpose of this is that an event based remote protocol server might
> still
> not yet have the `MSG_GETMSET` message from the client as a response, and
> it'd
> be bad practice (or catastrophic) to block the thread while the message
> arrives.
> This blocking (if any) should be handled further up, by the event loop.
>
>
> 3. Method run() split in two: run() and run_one()
> -------------------------------------------------
>
> The method run() calls run_one() in a loop, but we don't always want this
> behavior. The reason is, again, event based messaging might decide to start
> attending other clients on the same thread without blocking, thus an
> infinite
> while for these type of servers is not desired.
>
>
> 4. Abstract RemoteProtocol class added
> --------------------------------------
> RemoteServer extends from abstract RemoteProtocol, and only implements a
> few
> members needed by the protocol. This effectively separates the protocol
> from the
> connection and the server.
>
> RemoteProtocol with virtual abstract methods for:
>
> * get_message - Receives a new message (waiting when needed).
> * send_message - Sends a message to the client.
> * get_db - Reserves a database when the class needs to use it.
> * release_db - Releases the database when the class is not using it.
> * select_db - It's called when the protocol needs to select a new database
> to work on.
> * shutdown - It's called when the protocol requires to shutdown.
>
> Each command in the RemoteProtocol now do a two additional things:
>
> 1. Use get_db() to reserve and get a read-only or writable database.
> 2. Relese the reserved database using release_db()
>
> Implements RemoteProtocol on the RemoteServer:
>
> * get_db - Always returns the same database object (db or wdb)
> * release_db - Does nothing in this server (database is never released).
> * get_message and send_message - Call RemoteConnection methods.
>
>
> 5. RemoteProtocol implementation moved to remoteprotocol.cc
> -----------------------------------------------------------
>
> Added `remoteprotocol.cc` with the implementation of `RemoteProtocol` and
> header
> file remoteprotocol.h with added `RemoteProtocol` abstract class definition
> moved to `include/xapian/remoteprotocol.h`
>
> The reason for this is other projects might want to use the
> `RemoteProtocol`
> class for implementing a custom remote server. Xapiand is one of these
> projects.
> (https://github.com/Kronuz/Xapiand)
>
>
> Please, any comments and feedback would be appreciated.
>
> German M. Bravo (Kronuz)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20151219/1b222f8b/attachment.html>