[Xapian-discuss] Converting MySQL database to Xapian

[Michael Lewis]

I am looking for some guidance on converting a large MySQL database to Xapian. The current structure is that the database is broken up into 160 "sub-databases". There are 50,000 or so records in each stub database. Each record has content that I am full-text indexing. The average size of the text is about 59k characters. The database is broken up into sub-databases because the MySQL full-text handling is so abysmal. I use MySQL-Proxy and some Lua modules to make the searching and inserting transparent to the client programs. This data is written once and never (or once in a blue moon a record is deleted) modified. What I am looking to do is:

1. Create a xapian database on a remote server where each record's indexed text content would be a document (storing the document ID in the MySQL data record). I will then load the DB from the old MySQL server into the xapian server (eventually retiring the old server). It must remain in operation during the change-over.

2. Be able to access the xapian database via PHP and Omega from multiple clients (read only) on remote systems.

3. Insert new content on the server containing the xapian database. This insertion will take place on the server containing the xapian database.

Looking over the docs and googling suggests that I may need multiple xapian databases since the update time appears to be appreciable and I can't knock the db offline while updating (24/7 operation).

Omega will be run from a remote system.

Part of my problem is that I am not sure of how the remote backends are set up and do I need local stub DBs, which back-end to use, etc.

I would appreciate any help

Thanks,

Michael Lewis