[Xapian-discuss] Strange behavior with the Python bindings and threads

[jarrod roberson]

On 7/16/06, James Aylett <james-xapian at tartarus.org> wrote:
>
> On Sun, Jul 16, 2006 at 01:53:46AM +0100, Olly Betts wrote:
>
> > > The only thing we can think of is the SWIG Python bindings aren't
> releasing
> > > the GIL correctly or something?
> >
> > I don't know enough about them to say.  You could look at the generated
> > code (the C++ is python/xapian_wrap.cc and the Python is in
> > python/modern/ for Python 2.2 and later) or ask on the SWIG mailing
> > list.
>
> Python itself isn't great a concurrent threading, although three
> threads shouldn't generally cause significant problems.
>
> When you say you're completely IO bound, do you know *where* you are
> IO bound? What's your OS? What's your spindle layout? If you have the
> Xapian db on the same spindle as the files you're indexing, for
> instance, you may run into problems there (depending on the OS, cache
> settings, file system type etc.).


there is nothing CPU intensive going on,  I am only reading and writing 4k
of data to index.
there is something in the Python bindings making the Python threads
serialize on multiple CPU machines.

everything is on the same RAID5 array 5 250GB drives. It is a Dell 2850, it
is a "real" server machine, not some left over hardware.