[Xapian-discuss] How to index a lot of documents quickly

[Olly Betts]

On Thu, Mar 03, 2005 at 12:05:15AM +0000, Olly Betts wrote:
> At present, quartzcompact doesn't produce quite the same output from
> merging as it would when compacting a single file.  The issue is that
> the keys in 3 tables don't exactly sort in docid order, so the merging
> used doesn't write the keys in totally sorted order.  I'm just testing
> to see if this adversely affects the database size.  If it does, I can
> fix it, at the potential cost of a slightly slower merge.

I'm currently running the output of the gmane merge through
quartzcompact.  It's reduced the size of the record table by 43% (!), so
I think I need to address this (the postlist table is unsuprisingly
unchanged, and the value table isn't used so that will be too - it's
still working on the other 2).

Cheers,
    Olly