[Xapian-discuss] Optimal usage of xapian-compact for merging

[Henry C.]

On Mon, February 8, 2010 01:01, Olly Betts wrote:
>> Merging several hundred thousand dbs in a single invocation presents a
>> spot of bother :)
>
> What goes wrong?

You got me, I haven't even tried -- my arrogant assumption was that this
is simply not doable.   I can't even begin to get my head around invoking
"xapian-compact folder1 folder2 folder3 ... N dst" with N being rather
large (the command line will span many, many, many xterm pages).

...but, I just recalled why I defaulted to 50 or so dbs in a batch:  even
though I record index failure in a rdbms, it's always possible that a dud
will slip in, so when merging I check for failure (possibly due to a
corrupt/missing index).  Having a small batch makes it easy and efficient
to detect an error during merging, strip out the dud path, and try again.

Merging in one foul swoop will mean restarting the entire process if an
error occurs.

I'm just more comfortable merging in manageable chunks...  I feel I have
more control and have some flexibility.


h