[Xapian-discuss] Multithreaded read access
richard at lemurconsulting.com
Wed Aug 1 14:17:09 BST 2007
Lucian Nicolescu wrote:
> Inside the xmlrpc server I only open the xapian database for reading once.
> This causes requests to "stay in line" in order to get results and sometimes
> a one operation can take up to 1 second to complete (a time-taking operation
> includes text analisys and some more operations besides the actual Xapian
> interaction) thus delaying all the queued requests.
> I read lots of documents about xapian in mutithreaded environment and
> concluded that even for read access one cannot open the same database more
> than once and perform operations at the same time.
This is a misconception - for read access, you can open the same
database as many times as you like concurrently (well, as many as your
operating system will allow - eventually, you'll run out of file
handles). However, each of the Database objects you get by opening the
database must not be accessed concurrently. A typical model might be to
create N threads, and use N Database objects, one for each thread.
> I am using Xapian 0.9.6, Python bindings and Twisted Python XMLRPC package.
Just to note - Xapian 0.9.6 is rather old - the latest release is 1.0.2.
1.0.2 has the benefit that the Python bindings are set up to release
the GIL whenever a potentially long-running xapian operation is in
progress, which will give better performance in a concurrent environment.
More information about the Xapian-discuss