[Xapian-discuss] flintlock fork causes hang in Apache+Python+mod_python

[Deron Meranda]

On 8/28/07, Richard Boulton <richard at lemurconsulting.com> wrote:
> Deron Meranda wrote:
> > I am trying to use Xapian 1.0.2 with the Python SWIG
> > bindings withn an environment consisting of Apache
> > httpd with mod_python.  (not as a CGI)  Also this is
> > Linux.
> >
> > Whenever the python code attempts to open a database
> > the entire httpd process will hang indefinitely. [...]
>
> I think this is probably a slightly different problem which was
> identified on the list ...
>
> By default, mod-python uses a separate instance of the python
> interpreter for
> each mod-python application.  However, the Xapian python bindings do not
> know
> how to get and release control of the thread lock unless they are running in
> the "main" python interpreter; as a result, deadlocks can occur.

I bet that's it.  I did some more low-level system call tracing
and found the deadlock was actually hanging before the eventual
fork and lock calls, but immediately inside the call to
xapian.Database() or xapian.WritableDatabase().

I was thinking for a while it might be my strict SELinux
policies (my Apache and document indexer run under
different security contexts), but that all checked out okay.
Although it would be nice if xapian didn't create new
flintlock files with Unix permissions 600 (u=rw), but I
can work around that.

I use a threadless Apache MPM, meaning Apache is not
running multithreaded also implying my Python is single-
threaded as well.  But I DO use mod_python with multiple
interpreter instances inside each process.  Each httpd
process probably has 4 or 5 separate python interpreters
embedded in it at any given time.

> To avoid the problem, you need to set the "PythonInterpreter
> main_interpreter"
> option in the apache configuration section for all mod_python scripts
> which use
> xapian:

Hmmph, this is not an option for me.  In my setup I must
divide things into separate interpreters, so I don't have the
option of only using Xapian when in the main interpreter.

> this ensures that the xapian bindings are called from the
> main_interpreter, and can handle the thread locks appropriately.  An example
> configuration file using this option is available <a
> href="http://www.xapian.org/cgi-bin/bugzilla/attachment.cgi?id=121&action=view">here</a>.
>
> In the longer term, we hope to be able to resolve this problem, so that
> arbitrary python interpreters can be used; but this is quite a tricky
> problem
> to fix, and is relatively low priority.

Well, it's pretty high priority for me.  I'm pretty new to Xapian,
but perhaps I can help.  I've been pretty active in the mod_python
developer's community for a while.


> More details can be found in Xapian's
> bug tracker, as bug #185:
> http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=185

Thanks, I'll go look.

I guess then that this is a known bug, and this thread should
move over to the dev mailling list (which I guess I should
join now).

Deron Meranda