[Xapian-devel] About memory index/search in multithread program

[Olly Betts]

On Thu, Jun 19, 2014 at 10:52:13PM +0800, Yanxiong Lu wrote:
> Why xapian don't support memory  index/search ?
> 
> I know there is a method can create memory datebase, like this:
> Xapian::WritableDatabase db(Xapian::InMemory::open());

While the inmemory backend is handy for some things, in its current
incarnation some operations really aren't very time efficient, and it
doesn't even try to be very space efficient.

The longer term plan is to replace it, probably sharing code with the
disk based backends (which would make it fairly easy to do neat things
like migrating data from "inmemory" to disk).

> *But, if i use these in multithread program,  i need create many
> datebases!!*
> Xapian::WritableDatabase db1(Xapian::InMemory::open());  //used in thread1
> Xapian::WritableDatabase db2(Xapian::InMemory::open());  //used in thread2
> 
> because WritableDatabase object isn't thread-safe. And use lock is slowly.

Pushing those locks inside the Xapian library won't make things any
faster - likely just the opposite.  If the API user locks, they
can choose the granularity, while if Xapian locks, it has to lock
and unlock around every API method called.

I guess a lock-free, thread-safe inmemory backend implementation might
be possible, but realistically if you want one any time soon, you're
probably going to need to contribute most of the code for it.

> *So, is there some solutions that One database, many thread can use??*

The simplest solution currently is probably to create a RAM disk and use
a disk-based backend in it.

Cheers,
    Olly