[Xapian-devel] Available backends in xapian.h

[Olly Betts]

On Sun, May 08, 2005 at 04:35:58PM +0200, Francesco Garosi wrote:
> 1) path of Python modules, useful to install (like os.py):
> python -c "import os;print os.path.join(os.path.dirname(os.__file__), 'site-packages').replace(os.sep,'/')"

Hmm, I think I've found a better way:

python -c 'import os,distutils.sysconfig;print distutils.sysconfig.get_python_lib(1).replace(os.sep,"/")' 

The '1' means the directory for platform-specific libraries which seems
correct as we put _xapian.so here.  But on Debian at least it's the same
directory as the platform-independent one.

I think that should give the correct value on MS Windows.

I've put that in place in configure.  Also you can now override where
the python bindings get installed like so:

./configure PYTHON_LIB=/home/olly/my-site-python

which might be useful for installing your own private version on a box
where you don't have admin access.

> The Python prefix can also be modified using
> python -c "import os,sys;print sys.prefix.replace(os.sep,'/')".

We also use the prefix for finding the Python include files, which
I think we can do better using:

python -c 'import os,distutils.sysconfig;print distutils.sysconfig.get_python_inc().replace(os.sep,"/")'

I think that should also give the correct value on MS Windows.

If you could check these changes on mingw that'd be very helpful

> Regarding the need on Win32 to explicitly link the module against some "stub"
> static library that exports the symbols defined in pythonXY.dll (I think
> that it should be the same for other interpreters, like Tcl and PHP), I
> still didn't find a way to let Python tell us where the static libraries
> are installed. The use of MinGW to build Python modules is actually not
> completely "official". There are static libraries for MSVC in %PYTHON_PREFIX%\libs
> (notice the 's' ending), but this is not our case.

Isn't mingw ABI compatible with MSVC?  That was my understanding...

> I think that the fact that ld expects explicit parameters like "-lpython23
> -L$PYTHON_PREFIX/libs" is only a Win32 feature, as the DLL is actually an
> executable. I know that libtool should resolve dependencies on its own,
> but it looks like it cannot handle this here.

That's because libtool can only handle them if it knows about them.  So
either they need to be in the Makefile.am, or you need to link against
a libtool .la file (which describes dependencies).

> As far as i could see, this (and tweaking some variables in Makefile, that
> I'll report later) produces a regular _xapian.dll file, that Python for
> Win32 imports. But Win32 is really fussy, and it fails to find libxapian-5.dll
> as it is not in $PATH (as soon as it's able to import xapian.py, see below).
> If I modify $PATH in advance, _xapian.dll can be imported. Do you know if
> there is a way to know its location from whitin the build process? xapian-config
> does not report current installation or build prefix at the moment.

It would be easy enough to add 'xapian-config --prefix', though it may be
the wrong approach.  In the uninstalled case, reporting the build
directory as the prefix doesn't work properly as the libs aren't in a
lib subdirectory (for example), and anyway it's the wrong prefix to
build into anything after installing.  Also it's wrong to assume that
libs are in $prefix/lib as configure --libdir can change that (and at
least on Linux, it's common to have lib32 and lib64 nowadays).

Adding 'xapian-config --libdir' may be the answer, but we need to decide
if it should report "where the library is now" or "where the library is
or will be installed".

> I also noticed that there is a small problem with the SWIG generated xapian.py:
> the current python/Makefile.am, at line 53, says:
> 
> modern/xapian_wrap.cc modern/xapian_wrap.h modern/xapian.py: ../xapian.i
> util.i extra.i
> 	test -d modern || mkdir modern
> 	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow -modern \
> 	    -o modern/xapian_wrap.cc $(srcdir)/../xapian.i
> 
> olde/xapian_wrap.cc olde/xapian_wrap.h olde/xapian.py: ../xapian.i util.i
> extra.i
> 	test -d olde || mkdir olde
> 	$(SWIG) -I$(srcdir) @SWIG_FLAGS@ -c++ -python -shadow \
> 	    -o olde/xapian_wrap.cc $(srcdir)/../xapian.i
> 
> My SWIG (swig.exe is 1.3.24, while 1.3.22 is swig-1.3.22.exe as suggested
> by Olly) fails to create the .py file in the modern and olde subdirectories,
> and the build break.

This works for me.  Looking at the SWIG code it's trying to split
"modern/xapian_wrap.cc" on '\' to find the default outdir setting.

> This can be fixed by specifying -outdir <subdir> like
> this:

OK, passing -outdir won't change anything for UNIX, so I'll just add it.

> I also think that, provided the modification now in SVN, the lines with
> $(PYTHON_PATHSEP) should have double quotes around the right side of PYTHONPATH
> assignment, because a semicolon could be interpreted as an end-of-statement
> (line 3 and line 43 of python/Makefile.am):
> 
> TESTS_ENVIRONMENT = PYTHONPATH="@PYTHON_MODERN_OR_OLDE@$(PYTHON_PATHSEP).libs"
> $(PYTHON)

This had actually crossed my mind when I wasn't at the computer, but I'd
forgotten again so thanks for pointing it out.  I've added the quotes
now.

> These are small steps - if the modifications do not break the usual UNIX
> build system, but there are still the variables I had to tweak in the Makefile
> in xapian-bindings/python (by the way, I'm sorry but I only concentrated
> my tests to have a clean build of the Python module).

Don't worry.  We need to start somewhere, and some of the changes will
help other bindings anyway (like passing -no-undefined).

> The variables are:
> 
> 1) to allow explicit linking of libpython23.a:
> LIBS = -LC:/Development/python-2.3/libs -lpython23

OK, I've added some configure magic which should add this, but I'm
unable to test it.

> 2) instead of the usual UNIX directory layout:
> PYTHON_INC = C:/Development/python-2.3/include
> PYTHON_LIB = C:/Development/python-2.3/Lib/site-packages
> pylibdir = C:/Development/python-2.3/lib/site-packages

OK, this should no longer be needed with the changes I've just made.

> 3) to allow MinGW to #include "Python.h" (still not the UNIX layout):
> AM_CPPFLAGS = -IC:/Development/python-2.3/include

Nor this.

> I am afraid that to some extent a particular logic has to be implemented
> to modify these variables, and that at least temporarily, a developer should
> be warned that $PATH has to include the path to a valid and corresponding
> libxapian-X.dll in order to build the module. The installation (and a binary
> distribution) of the module should also include this library to be functional,
> and for this purpose the directory found at point 2 above (the one with
> lib-dynload) can be useful.

I'm not sure how best to address this, so I'll leave it for the time
being.  I'll release 0.9.0 and then we can get back to this.

> I hope that what I found is not completely useless...

It was very helpful!

Cheers,
    Olly