[Xapian-tickets] [Xapian] #424: Magic filter limits are a bad idea

[Xapian]

#424: Magic filter limits are a bad idea
--------------------+-------------------------------------------------------
 Reporter:  chrisc  |       Owner:  olly
     Type:  defect  |      Status:  new 
 Priority:  normal  |   Milestone:      
Component:  Omega   |     Version:      
 Severity:  normal  |    Keywords:      
Blockedby:          |    Platform:  All 
 Blocking:          |  
--------------------+-------------------------------------------------------

Comment(by richard):

 Our desire is for omindex to work out-of-the-box with sensible default
 behaviour - forcing all users to perform custom configuration to prevent
 denial of service issues, and issues with omindex failing to finish due to
 some bad filters, isn't in line with this aim.  This applies whether the
 configuration required is setting ulimit, or setting up a configuration
 file.  The default process memory ulimit on my (ubuntu) system is
 "unlimited", so your suggested patch would force me to perform such a
 configuration step, for example.

 Therefore, if we were to start using a configuration option, it would need
 to default to using some kind of limit, and the current behaviour is
 probably the default we'd want to use; I haven't heard or thought of a
 better default than using a (large) proportion of the available memory on
 a system.

 As Olly said, we're not necessarily against adding some configuration in
 this area, but if there's no actual situation in which users would benefit
 from adjusting the configuration, we'd just be adding code bloat - so we
 need a justification for adding configurability in the form of a
 description of a situation which is helped by the configurability.

 I was not aware of this issue in 1.0.7 rendering omindex practically
 useless on Linux - there doesn't seem to be a ticket about this in this
 Trac instance, and I can't find anything on the subject with a quick
 google.  Could you provide a reference for this, so we're not missing
 relevant information, please?  I use omindex on Linux regularly and it
 works for the files I'm passing it; perhaps I'm not using the same filters
 as you, though.

 The limits were added in response to ticket #111 - as Olly said, this was
 in response to real problems experienced by users.  Setting ulimit is
 another mechanism for limiting resource usage, but would require users to
 do this explicitly, and anyway I'm not convinced we'd want the limit for
 the main omindex process to be the same as that for the filter programs -
 omindex can end up using a significant amount of memory while indexing,
 without that necessarily representing a problem.

 I did find ticket #358 (and the linked http://bugs.debian.org/cgi-
 bin/bugreport.cgi?bug=548987 ) which indicates that omindex used to use
 _SC_AVPHYS_PAGES instead of _SC_PHYS_PAGES, and this did cause a problem -
 however, it's been changed to use _SC_PHYS_PAGES, and I can't see any
 mention of this not working correctly since 1.0.17 where the fix was
 applied.

 If there's a real (or perhaps even hypothetical) situation in which the
 current code fails to work well, please describe it to us so we can
 improve the design and code accordingly.

-- 
Ticket URL: <http://trac.xapian.org/ticket/424#comment:3>
Xapian <http://xapian.org/>
Xapian