How to make database build threaded?

[Olly Betts]

On Fri, Sep 14, 2018 at 06:09:02PM +0200, Franco Martelli wrote:
> Thank for your answer, briefly it's No:
> 
> On 14/09/2018 at 09:30, Jean-Francois Dockes wrote:
> > The Xapian library index updating code is not designed for
> > multi-threading and must stay protected from multiple accesses.

That's perhaps slightly misleading - it's really the data that needs
protecting, not the code as such.

Xapian doesn't attempt to protect any of its objects from concurrent
access, but you can use different objects from the same class
concurrently.  So for example you can have two different databases open
in two different threads and index to them from concurrently.

You can then either just search those databases together or use
Xapian::Database::compact() (or the xapian-compact command line tool) to
merge the databases once built.

Or if your document processing stage is relatively slow and you're
using Xapian 1.4.6 or newer, you can prepare Xapian::Document objects
in worker threads and then use the new support for C++11 move semantics
to cleanly and efficiently hand them to a single indexer thread to
add to the database.  Before 1.4.6 you'd have had to use some sort of
locking to do that safely.

Cheers,
    Olly