[Xapian-tickets] [Xapian] #358: Omega: omindex eating up all available physical memory
Xapian
nobody at xapian.org
Tue Apr 21 15:03:08 BST 2009
#358: Omega: omindex eating up all available physical memory
----------------------+-----------------------------------------------------
Reporter: evoisard | Owner: olly
Type: defect | Status: new
Priority: normal | Milestone:
Component: Omega | Version:
Severity: normal | Keywords:
Blockedby: | Platform: Linux
Blocking: |
----------------------+-----------------------------------------------------
I'm having a recurring problem with Omega's indexing with omindex.
When I run omindex, sometimes it is like if it was missing to recognize
the extension of some .doc and .pdf files and it skips them with an
"Unknown extension ... - skipping" message.
In the same run, omindex is otherwise perfectly able to index other files
with same extensions.
If I manually run antiword on a .doc file that failed previously, it
works.
If I narrow down the directory structure to make the recursion and
indexing lighter, and then I run omindex, it works with files that failed
previously.
It never seems to fail with html and plain text files (built-in formats)
Each time a failure occurs and a file is skipped, a kernel error like the
following one is recorded in /var/log/messages:
{{{Apr 21 14:10:12 zen kernel: sh[4153]: segfault at ffffffffffffffff rip
00002ac7e7c4581f rsp 00007fffc3452de0 error 4}}}
As reported by some other users who had same problem, it can be due to my
system running low on memory and omindex not being able to run the
external converter.
I ran omindex while checking the system's memory usage. The system
(SLES10) has 1GB of RAM. Approx two thirds of that was used by other
processes. Over the free third, omindex gradually took up all until 10MB
remained free. At this point, memory usage stabilized and failures began
to occur. Remember that it doesn't fail on every subsequent .doc or .pdf,
but only on some of them.
--- Before running omindex:
{{{
top - 13:44:50 up 187 days, 21:46, 6 users, load average: 2.02, 2.03,
2.08
Tasks: 149 total, 2 running, 147 sleeping, 0 stopped, 0 zombie
Cpu(s): 50.0%us, 0.0%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 1027164k total, 654256k used, 372908k free, 7760k buffers
Swap: 4200988k total, 258284k used, 3942704k free, 404760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3268 root 34 19 261m 4656 2440 S 100 0.5 270524:02 zmd
1 root 16 0 796 72 40 S 0 0.0 0:00.96 init
2 root RT 0 0 0 0 S 0 0.0 0:00.18 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.14 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
}}}
--- Beginning (omindex using 9860kB or less than 1% of RAM):
{{{
top - 13:45:35 up 187 days, 21:47, 6 users, load average: 2.16, 2.05,
2.09
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
Cpu(s): 63.7%us, 6.5%sy, 0.0%ni, 29.4%id, 0.0%wa, 0.0%hi, 0.5%si,
0.0%st
Mem: 1027164k total, 678548k used, 348616k free, 8224k buffers
Swap: 4200988k total, 258284k used, 3942704k free, 421772k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3268 root 34 19 261m 4656 2440 S 93 0.5 270524:47 zmd
829 root 17 0 16864 6520 1788 D 20 0.6 0:00.64 omindex
28316 root 10 -5 0 0 0 S 1 0.0 1:12.45 cifsd
31328 root 16 0 5656 1260 876 R 1 0.1 0:14.33 top
1 root 16 0 796 72 40 S 0 0.0 0:00.96 init
[evoisard at zen]/home/evoisard > date ; ps aux | grep omindex
Tue Apr 21 13:45:40 CEST 2009
root 829 8.7 0.9 20052 9860 pts/5 D+ 13:45 0:01 \
/usr/local/bin/omindex --db /srv/xapian/test --follow --url /docs/test/
/srv/xapian/targets/test
}}}
--- During runtime, still working fine (omindex using 103200 kB or 10% of
RAM):
{{{
top - 13:56:14 up 187 days, 21:58, 6 users, load average: 3.06, 2.98,
2.58
Tasks: 153 total, 1 running, 152 sleeping, 0 stopped, 0 zombie
Cpu(s): 60.7%us, 3.0%sy, 0.0%ni, 35.8%id, 0.0%wa, 0.0%hi, 0.5%si,
0.0%st
Mem: 1027164k total, 824360k used, 202804k free, 10340k buffers
Swap: 4200988k total, 258284k used, 3942704k free, 464760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3268 root 34 19 261m 4656 2440 S 99 0.5 270535:07 zmd
829 root 17 0 110m 100m 1820 S 25 10.0 0:55.32 omindex
31328 root 16 0 5656 1260 876 R 1 0.1 0:16.99 top
1 root 16 0 796 72 40 S 0 0.0 0:00.96 init
2 root RT 0 0 0 0 S 0 0.0 0:00.18 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
[evoisard at zen]/home/evoisard > date ; ps aux | grep omindex
Tue Apr 21 13:56:17 CEST 2009
root 829 8.5 10.0 113396 103200 pts/5 D+ 13:45 0:55 \
/usr/local/bin/omindex --db /srv/xapian/test --follow --url /docs/test/
/srv/xapian/targets/test
}}}
--- Close to the end, docs skipping and segfaults occuring (omindex using
369340kB or 36% of RAM):
{{{
top - 14:10:23 up 187 days, 22:12, 6 users, load average: 3.19, 3.22,
2.96
Tasks: 152 total, 2 running, 150 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.6%us, 1.5%sy, 0.0%ni, 0.0%id, 3.0%wa, 0.0%hi, 1.0%si,
0.0%st
Mem: 1027164k total, 1017024k used, 10140k free, 996k buffers
Swap: 4200988k total, 258284k used, 3942704k free, 401204k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3268 root 34 19 261m 4656 2440 S 100 0.5 270549:04 zmd
829 root 18 0 370m 360m 1920 D 93 36.0 5:05.18 omindex
154 root 15 0 0 0 0 S 1 0.0 0:08.67 kswapd0
31328 root 16 0 5656 1260 876 R 1 0.1 0:20.52 top
1 root 16 0 796 72 40 S 0 0.0 0:00.96 init
2 root RT 0 0 0 0 S 0 0.0 0:00.18 migration/0
[evoisard at zen]/home/evoisard > date ; ps aux | grep omindex
Tue Apr 21 14:10:28 CEST 2009
root 829 20.5 35.9 379324 369340 pts/5 D+ 13:45 5:08 \
/usr/local/bin/omindex --db /srv/xapian/test --follow --url /docs/test/
/srv/xapian/targets/test
}}}
When omindex terminates, all reserved resources are freed.
So, it looks like omindex somehow is not releasing all the runtime memory
it is using. Sure I could add more memory to the system, but would then
omindex not eat up all the extra memory too, or would it not have same
problem again if the directories to index increase in size..
I don't know if this memory is required for handling the database itself
or if it's used for the runtime and the filtering/indexing jobs.
I don't know if this behavior should be considered a bug or not, or if the
process could be optimized. I let Xapian masters decide...
Anyway, many thanks for the wonderful work! Eric
--
Ticket URL: <http://trac.xapian.org/ticket/358>
Xapian <http://xapian.org/>
Xapian
More information about the Xapian-tickets
mailing list