[Xapian-tickets] [Xapian] #424: Magic filter limits are a bad idea

[Xapian]

#424: Magic filter limits are a bad idea
--------------------+-------------------------------------------------------
 Reporter:  chrisc  |        Owner:  olly    
     Type:  defect  |       Status:  reopened
 Priority:  normal  |    Milestone:  1.2.4   
Component:  Omega   |      Version:          
 Severity:  normal  |   Resolution:          
 Keywords:          |    Blockedby:          
 Platform:  All     |     Blocking:          
--------------------+-------------------------------------------------------
Changes (by chrisc):

  * status:  closed => reopened
  * resolution:  fixed =>


Comment:

 The issue in 1.0.7 was a real usage condition where a busy system with
 4GBs of RAM running a 64-bit kernel, building a very large index (process
 resident size was around 2GB) would fail once the 'free memory' dropped
 down to about 120MBs - the AS limit would prevent fork() from building a
 valid process (minimum process space is about 128MB) resulting in further
 indexing failing.  Part of this was caused by the incorrect use of
 RLIMIT_AS (which should never be used in such a clumsy way), but the main
 problem was the sliding memory limit.

 Likewise, on similarly large memory systems, 7/8th physical is FAR TOO
 LARGE a limit to be useful - you've already done significant damage if a
 process is allowed to run-away long enough to consume that much memory.

 I accept that you need to stop the tools from running away, but you need
 to do so in a way with reasonable, controllable limits, not some magic
 number that you think is fine, and is only safe for the limited
 environment in which you test your code.

-- 
Ticket URL: <http://trac.xapian.org/ticket/424#comment:6>
Xapian <http://xapian.org/>
Xapian