[Xapian-discuss] too slow when create index

[Olly Betts]

On Fri, Aug 31, 2012 at 10:56:09AM +0800, Eric Yue wrote:
> I am create index for some files,in my program,a document is  a line in a
> file.   i create index for very lines in a file.  is there any method to
> speed up this ??????

You don't give us much information about how you're indexing, so it's
hard to give specific advice, but in general increasing the batch size
will make indexing go faster - provided you have enough memory.  The
default is to auto-commit every 10000 document changes, which is fairly
conservative for modern hardware.  And if each line in a file is a
document, then your documents are presumably very small so you should be
able to raise this a lot.

To set it, you just set XAPIAN_FLUSH_THRESHOLD in the environment (and
make sure it is exported) - e.g.:

env XAPIAN_FLUSH_THRESHOLD=1000000 ./my-indexer

Cheers,
    Olly