[Xapian-devel] Available backends in xapian.h

Francesco Garosi franz.g at tin.it
Sun May 8 15:35:58 BST 2005


>There is an easy way:

there are in fact some almost easy ways to discover more paths in a Python
installation, relying on the __file__ property that all non-builtin modules
have. I think that this should work on all platforms supported by Python:

1) path of Python modules, useful to install (like os.py):
python -c "import os;print os.path.join(os.path.dirname(os.__file__), 'site-packages').replace(os.sep,

On my Windows XP installation it reports:

On my "unusual" Solaris box (homebrewed Python 2.3):

On a standard RH 8.0 (with Python 2.2):

2) path of binary modules (like zlib.pyd, usually not mandatory to know):
python -c "import os,zlib;print os.path.dirname(zlib.__file__).replace(os.sep,

On Win32 it says:

On Solaris:

On RH 8.0:

The path at point 2) is not an usual place to install third party modules:
both Python and binary modules usually go in .../site-packages.

I used os and zlib as modules because they are mandatory in (I bet) every
Python installation: from Python 2.3 on Python also uses zlib.so (or zlib.pyd
on Win32) to import modules from zip files. The Python prefix can also be
modified using python -c "import os,sys;print sys.prefix.replace(os.sep,

Regarding the need on Win32 to explicitly link the module against some "stub"
static library that exports the symbols defined in pythonXY.dll (I think
that it should be the same for other interpreters, like Tcl and PHP), I
still didn't find a way to let Python tell us where the static libraries
are installed. The use of MinGW to build Python modules is actually not
completely "official". There are static libraries for MSVC in %PYTHON_PREFIX%\libs
(notice the 's' ending), but this is not our case. The almost unofficial
MinGW-based module build system still looks for static (possibly stub) libraries
in the aforementioned directory, and the official documentation says that
the library libpythonXY.a (obtained in a tricky way) has to be installed
in %PYTHON_PREFIX%\libs.

I think that the fact that ld expects explicit parameters like "-lpython23
-L$PYTHON_PREFIX/libs" is only a Win32 feature, as the DLL is actually an
executable. I know that libtool should resolve dependencies on its own,
but it looks like it cannot handle this here.

As far as i could see, this (and tweaking some variables in Makefile, that
I'll report later) produces a regular _xapian.dll file, that Python for
Win32 imports. But Win32 is really fussy, and it fails to find libxapian-5.dll
as it is not in $PATH (as soon as it's able to import xapian.py, see below).
If I modify $PATH in advance, _xapian.dll can be imported. Do you know if
there is a way to know its location from whitin the build process? xapian-config
does not report current installation or build prefix at the moment.

I also noticed that there is a small problem with the SWIG generated xapian.py:
the current python/Makefile.am, at line 53, says:

modern/xapian_wrap.cc modern/xapian_wrap.h modern/xapian.py: ../xapian.i
util.i extra.i
	test -d modern || mkdir modern
	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow -modern \
	    -o modern/xapian_wrap.cc $(srcdir)/../xapian.i

olde/xapian_wrap.cc olde/xapian_wrap.h olde/xapian.py: ../xapian.i util.i
	test -d olde || mkdir olde
	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow \
	    -o olde/xapian_wrap.cc $(srcdir)/../xapian.i

My SWIG (swig.exe is 1.3.24, while 1.3.22 is swig-1.3.22.exe as suggested
by Olly) fails to create the .py file in the modern and olde subdirectories,
and the build break. This can be fixed by specifying -outdir <subdir> like

modern/xapian_wrap.cc modern/xapian_wrap.h modern/xapian.py: ../xapian.i
util.i extra.i
	test -d modern || mkdir modern
	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow -modern \
	    -outdir modern -o modern/xapian_wrap.cc $(srcdir)/../xapian.i

olde/xapian_wrap.cc olde/xapian_wrap.h olde/xapian.py: ../xapian.i util.i
	test -d olde || mkdir olde
	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow \
	    -outdir olde -o olde/xapian_wrap.cc $(srcdir)/../xapian.i

I also think that, provided the modification now in SVN, the lines with
$(PYTHON_PATHSEP) should have double quotes around the right side of PYTHONPATH
assignment, because a semicolon could be interpreted as an end-of-statement
(line 3 and line 43 of python/Makefile.am):


xapian.pyc: xapian.py
	PYTHONPATH=".$(PYTHON_PATHSEP).libs" $(PYTHON) -c "import xapian"

This would not be noticed in an UNIX installation because there $(PYTHON_PATHSEP)
is still a colon.

These are small steps - if the modifications do not break the usual UNIX
build system, but there are still the variables I had to tweak in the Makefile
in xapian-bindings/python (by the way, I'm sorry but I only concentrated
my tests to have a clean build of the Python module). The variables are:

1) to allow explicit linking of libpython23.a:
LIBS = -LC:/Development/python-2.3/libs -lpython23

2) instead of the usual UNIX directory layout:
PYTHON_INC = C:/Development/python-2.3/include
PYTHON_LIB = C:/Development/python-2.3/Lib/site-packages
pylibdir = C:/Development/python-2.3/lib/site-packages

3) to allow MinGW to #include "Python.h" (still not the UNIX layout):
AM_CPPFLAGS = -IC:/Development/python-2.3/include

I am afraid that to some extent a particular logic has to be implemented
to modify these variables, and that at least temporarily, a developer should
be warned that $PATH has to include the path to a valid and corresponding
libxapian-X.dll in order to build the module. The installation (and a binary
distribution) of the module should also include this library to be functional,
and for this purpose the directory found at point 2 above (the one with
lib-dynload) can be useful.

I hope that what I found is not completely useless...



More information about the Xapian-devel mailing list