[Xapian-discuss] Xapian under 360 MB VPS

[Richard Boulton]

2010/1/21 Marlon Baculio <mbaculio at hotmail.com>:
> I'm used to multithreaded programming so I would tend to lean on that. I
> assume there are no file locking necessary in Xapian's read mode, or if
> there is, it's thread-safe? (I remember for Solaris, fcntl-based file
> locking doesn't work if you have multiple file descriptors to the same file
> from multiple threads (i.e. close() and lockf() don't mix). I did not
> encounter the same problem for Linux' flock() though.)

There is no file locking in Xapian's read mode at present (though it's
likely to be added at some point).  I think you can rely on us making
it thread safe if and when we add it, though - or if we can't do that
efficiently, documenting what guarantees the locking does provide.  We
went to great effort to ensure that the locking for writable databases
is threadsafe.

> Also, to service lots of concurrent searches, the way to go (aside from
> throwing more hardware) is to increase number of processes/threads in the
> pool with their own reader as Xapian is not asynchronous. Is that correct?

Xapian is not asynchronous, yes.

As to the rest of your question: possibly.  It depends on how big your
database is, really.  With this amount of memory, it's probably quite
easy for you to become IO bound, in which case running multiple
concurrent searches won't get the total work done much faster, and can
increase the latency for individual queries massively (complex queries
can end up waiting while lots of other queries do disk accesses).  You
might be best keeping the pool fairly small, and just queuing up
queries if the pool is all busy.

> (I might be prematurely optimizing here as I'm sure a $19 VPS hosting for
> 360MB will be a little tight for thousands of users, but I'm trying to get
> my money's worth :-)

If you run some experiments to see what profile you get, do share your
findings here; I'm sure others would be interested.

-- 
Richard