Xapian discuss - Aug 2007 - flintlock fork causes hang in Apache+Python+mod_python

I am trying to use Xapian 1.0.2 with the Python SWIG
bindings withn an environment consisting of Apache
httpd with mod_python.  (not as a CGI)  Also this is
Linux.

Whenever the python code attempts to open a database
the entire httpd process will hang indefinitely.  The
python bindings work outside of the apache/mod_python
environment.
>From the best I can tell the hang occurs in FlintLock:lock()
method.  I can see that the httpd process has forked off
a child using the "/bin/cat" executable, and the httpd process
(which has the embedded Python interpreter) is hung on
a FUTEX_WAIT system call).

While this process is hung, then attempts to open the
same database outside of Apache sometimes result
in an error saying that the database lock could not
be obtained because it is held by another process.

Has anybody experienced this problem?  I've never
seen a process locking mechanism before which uses
this fork/cat style mechanism.  Is it possible that this
could be confusing the python interpreter GIL or
something?

Deron Meranda

Deron Meranda wrote:
> I am trying to use Xapian 1.0.2 with the Python SWIG
> bindings withn an environment consisting of Apache
> httpd with mod_python.  (not as a CGI)  Also this is
> Linux.
> 
> Whenever the python code attempts to open a database
> the entire httpd process will hang indefinitely.  The
> python bindings work outside of the apache/mod_python
> environment.
I think this is probably a slightly different problem which was 
identified on the list (or maybe it was just on the bug tracker) a 
little while ago.  I added a piece of documentation on how to work 
around it to the SVN tree a little while ago, but it's not in a released 
version yet.


So you don't have to check out SVN, the documentation I wrote is as follows:

There is an issue with python's threading support and mod-python,
which can cause problems when using the Xapian Python bindings
from mod-python.  If you find that your mod-python application
is hanging whenever a Xapian method is called, you have probably
run into this problem.

By default, mod-python uses a separate instance of the python 
interpreter for
each mod-python application.  However, the Xapian python bindings do not 
know
how to get and release control of the thread lock unless they are running in
the "main" python interpreter; as a result, deadlocks can occur.

To avoid the problem, you need to set the "PythonInterpreter 
main_interpreter"
option in the apache configuration section for all mod_python scripts 
which use
xapian: this ensures that the xapian bindings are called from the
main_interpreter, and can handle the thread locks appropriately.  An example
configuration file using this option is available <a
href="http://www.xapian.org/cgi-bin/bugzilla/attachment.cgi?id=121&action=view">here</a>.

In the longer term, we hope to be able to resolve this problem, so that
arbitrary python interpreters can be used; but this is quite a tricky 
problem
to fix, and is relatively low priority.  More details can be found in 
Xapian's
bug tracker, as bug #185:
http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=185

-- 
Richard