[Xapian-discuss] Python bindings not freeing memory during indexing

[Richard Boulton]

EJ Johnson wrote:
> Thanks for taking the time to review my indexer Richard.  I don't have 
> an easy way to send you sample documents, but can describe them.
<snip>
> So, I guess the only thing you can really do (since you can't really 
> test the code) is to tell me if I'm doing something stupid with the API.

I've looked through the code, and it all looks perfectly sensible to me.

> I'm using Python 2.4.3 that ships by default on Ubuntu 6.06 LTS server 
> edition (dapper).

That should be fine.

> As far as the errors, the first set of errors I was seeing was a failed 
> call to st95_malloc (or something like that) that seemed to be thrown 
> from the SWIG bindings.  I'd have to re-run my older code and let it run 
> for 120minutes to regenerate the error.

Have you re-tried with the 1.0.2 release?  The best way forward, I 
think, is to re-run the test, with the workarounds removed so that the 
error occurs (if it still does with 1.0.2), and to keep the full output 
so we can pick through it.

Looking into why there are different document sizes with and without 
your workaround would be useful, too.  You could do this by comparing 
some random documents with the delve tool in the indexes built with and 
without the workaround.

One other question - is the process you're running the indexer in using 
multiple (python) threads?  There was a bug in 1.0.1 which could have 
caused corruption in this case - this is fixed in 1.0.2.

-- 
Richard