[Xapian-discuss] Re: Big process using Xapian

[James Aylett]

On Fri, Feb 02, 2007 at 07:17:04AM -0800, Rafael SDM Sierra wrote:

> >[1] - 736M   694M biord  182:27  2.15% python
>
> I change from 1000 to 10000 the xapian flush threshold, and the process
> become bigger oO...
> 
> 2008M  1504M swread  10:36  0.00% python
> 
> It's all that I have of memory (2GB), my swap is in use now...

Some systems cannot free main memory from the process back to the
operating system, even if it is unused within the process.

The larger the flush threshold, the more data has to be held in
memory, so this might explain what you're seeing. You need to keep the
flush threshold low enough that you have head space above the process
within core (physical memory), to enable some sort of write-behind
buffer at the operator system level (although I *believe* that with
Xapian's access pattern while writing it won't make a huge amount of
difference with most modern disks, but that's getting into very dicey
territory trying to claim that with confidence :-).

(This may not actually be your problem in this case, of course...)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org