[Xapian-discuss] PHP indexing, what's the PHP method for indexscript

[James Aylett]

On Wed, Jan 16, 2008 at 09:58:11AM -0800, athlon athlonf wrote:

> >Load 5 suggests something's wrong, because dbi2omega and scriptindex
> >are both linear processes. Are you running several instances in
> >parallel in some way?
> 
> it usually starts off fairly low, but then after half an hour of so,
> it will reach load 5 constantly.

I'd guess at a memory problem then, with processes blocking on VM
I/O. But that's really a guess.

Load is an indication of the length of the run queue, ie the number of
processes trying to get access to the processor at any one point in
time; so there's something going on beyond scriptindex itself
there. If you want to figure out what's going on, I'd recommend
pulling snmp data out of the system and graphing it (probably using
something like cacti) -- you'll see things like memory usage over time
that way, and it'll be obvious if (for instance) the load shoots up
when it hits a certain amount of free memory or something.

> I'm only doing one scriptindex at a time, but it's fairly
> complicated indexingscript I guess. And of course, the inputfiles
> are huge, mostly beyond 2GB.

scriptindex only pulls a line at a time from the input files, so that
won't matter per se. You may be running into issues with Xapian not
flushing to disk enough - that again 

> topic_title : unhtml weight=10 field=title index=Z index

What is the index=Z intended to do?

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org