[Xapian-discuss] XAPIAN_FLUSH_THRESHOLD

[Olly Betts]

On Thu, Jul 16, 2009 at 02:30:39AM +0930, Frank John Bruzzaniti wrote:
> Am I right in saying that for my setup I should be doing export 
> XAPIAN_FLUSH_THRESHOLD=1000 because:
> 
> 1000 documents * 2MB doc size = 2gig of memory required before a flush 
> to disk?

That's a bit simplistic, but probably a reasonable starting point.

What is stored in memory are changes to the postlist and spelling tables
- changes to other tables are written out (but not switched live).  For
the postlist table, there are the terms which have changed, the docids
and wdfs for those changes, and data structure overheads.  That's
probably going to come out significantly smaller than the raw text size
of the documents, so you can probably go higher than 1000.

Cheers,
    Olly