[Xapian-discuss] Python bindings not freeing memory during indexing

[Olly Betts]

On Sun, Jul 08, 2007 at 08:45:05PM -0500, EJ Johnson wrote:
> The 1.0.2 release isn't doesn't appear to be available for Dapper

That's now fixed (the 1.0.2 packages were in place, but I hadn't
uploaded the new index files).

> => indexing 526 more tickets (total:94922)
>   File "./ticketloader.py", line 146, in ?
>     docid = db.add_document(doc)
> RuntimeError: St9bad_alloc

That's not a failed function call - it's the result of called e.what()
when e is of type std::bad_alloc (I'm not quite sure why it has an
"St9" prefix - it's not just C++ name mangling).

So it looks like a call to new is failing with bad_alloc.  That probably
means you've run out of memory, or perhaps something has trashed the
heap.  You could try setting (and exporting) MALLOC_CHECK_=2 which will
tell glibc's malloc implementation validate the heap and call abort() if
it detects corruption.

Either way, it would be good if you could find some way to reproduce it
for us.

> One point of interest is that for my hand-compiled 1.0.2 version,
> it honored the XAPIAN_FLUSH_THRESHOLD environment variable (which I
> had set to 1000 for the run that produced these errors.  In the 1.0.1
> release for Dapper from xapian.org, it would ignore the env var
> which led to one of my workarounds to call flush() on my own.

Others have reported similar issues, but I've been unable to reproduce
them myself and nobody has so far prepared a recipe which demonstrates
it.

> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, [...]

Please don't send mail with such notices to public mailing lists.
The whole purpose of a public mailing list is to do many of the things
it says we can't.  Essentially it imposes a potential legal liability
on people you're asking to help you for free.

If you're unable to prevent your work email system from appending them,
sign up for a free webmail account and use that for mailing lists.  Or
find a new job with a saner employer.

Cheers,
    Olly