[Xapian-tickets] [Xapian] #664: omindex hangs on indexing 10G database

[Xapian]

#664: omindex hangs on indexing 10G database
----------------------+----------------------------
 Reporter:  hjohanns  |             Owner:  olly
     Type:  defect    |            Status:  closed
 Priority:  normal    |         Milestone:
Component:  Omega     |           Version:  1.2.16
 Severity:  normal    |        Resolution:  notabug
 Keywords:  hang      |        Blocked By:
 Blocking:            |  Operating System:  Linux
----------------------+----------------------------
Changes (by hjohanns):

 * status:  new => closed
 * resolution:   => notabug


Old description:

> I have been launching omindex on a 10G database. It hangs at a certain
> point without warning.
> There is nothing special in the log, and it does not hang always at the
> same point in the data.
> At that moment the process is taking 3.5G of memory, of which 2.76G
> resident. It slightly changes, but remains there for hours.
> How can I troubleshoot the issue?
> thanks for any help or suggestions, and have a great day,
> hubert
>
> Platform: Linux Ubuntu 14.04.1
> Package xapian-omega in Version: 1.2.16-1

New description:

 I have been launching omindex on a 10G database. It hangs at a certain
 point without warning.
 There is nothing special in the log, and it does not hang always at the
 same point in the data.
 At that moment the process is taking 3.5G of memory, of which 2.76G
 resident. It slightly changes, but remains there for hours.
 How can I troubleshoot the issue?
 thanks for any help or suggestions, and have a great day,
 hubert

 Platform: Linux Ubuntu 14.04.1
 Package xapian-omega in Version: 1.2.16-1

 Closed as user error. See comment for details

--

Comment:

 Closing the ticket as user error.
 I was saving a dirty file, binary, but qualified as html to xapian. When
 indexing it, and seeing the random bytes, vocabulary for xapian would
 explode and fill the memory.
 Key to analysis was to use delve and see that the vocabulary was largely
 composed of random binary words.
 Solution is to cancel those files from indexing, or qualify their type
 properly.

--
Ticket URL: <http://trac.xapian.org/ticket/664#comment:4>
Xapian <http://xapian.org/>
Xapian