[Xapian-discuss] Strange behavior with the Python bindings and threads

[James Aylett]

On Mon, Jul 17, 2006 at 11:21:16AM -0400, jarrod roberson wrote:

> >Python itself isn't great a concurrent threading, although three
> >threads shouldn't generally cause significant problems.
> >
> >When you say you're completely IO bound, do you know *where* you are
> >IO bound? What's your OS? What's your spindle layout? If you have the
> >Xapian db on the same spindle as the files you're indexing, for
> >instance, you may run into problems there (depending on the OS, cache
> >settings, file system type etc.).
> 
> there is nothing CPU intensive going on,  I am only reading and writing 4k
> of data to index.

What size are the documents you are reading (apologies if you've
mentioned this before)? And how many?

> everything is on the same RAID5 array 5 250GB drives. It is a Dell 2850, it
> is a "real" server machine, not some left over hardware.

I'll suppress an automatic snide remark about Dells :-)

What OS (I'm guessing a linux kernel, but which version at least would
be helpful)? How is the array attached to the computer? Is the RAID in
hardware or software? Is there a separate memory cache for the array?

> there is something in the Python bindings making the Python threads
> serialize on multiple CPU machines.

That doesn't surprise me. The bindings are heavily dependent on how
SWIG chooses to implement things. No idea quite *why* one of the
Python central locks is causing problems, or what we can do about it -
Richard has a bit more knowledge of Python threading than I do.

Does anyone know if the concurrency issue applies to Jython? I've
wondered for a while, but never had a chance to look into it.

James

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org