[Xapian-discuss] PHP indexing, what's the PHP method for indexscript

[athlon athlonf]

Actually I'm already graphing my servers, but haven't really investigate on the load due to the scriptindex as I was more devving atm.
I've just taken a look and it seems not to be a memory problem. During the load spikes, memory went from 20MB to 1GB, but there was still 2GB left (albleit cached).

I'd suspect it to be an IO load.

Hmm... the index=Z... no idea really. A copyandpaste-error i think. I remembered that when I delved my first xapian-db, i see that "content" which will be searched had the prefix Z, so...

----- Original Message ----
From: James Aylett <james-xapian at tartarus.org>
To: xapian-discuss at lists.xapian.org
Sent: Wednesday, January 16, 2008 7:19:25 PM
Subject: Re: [Xapian-discuss] PHP indexing, what's the PHP method for indexscript


On Wed, Jan 16, 2008 at 09:58:11AM -0800, athlon athlonf wrote:

> >Load 5 suggests something's wrong, because dbi2omega and scriptindex
> >are both linear processes. Are you running several instances in
> >parallel in some way?
> 
> it usually starts off fairly low, but then after half an hour of so,
> it will reach load 5 constantly.

I'd guess at a memory problem then, with processes blocking on VM
I/O. But that's really a guess.

Load is an indication of the length of the run queue, ie the number of
processes trying to get access to the processor at any one point in
time; so there's something going on beyond scriptindex itself
there. If you want to figure out what's going on, I'd recommend
pulling snmp data out of the system and graphing it (probably using
something like cacti) -- you'll see things like memory usage over time
that way, and it'll be obvious if (for instance) the load shoots up
when it hits a certain amount of free memory or something.

> I'm only doing one scriptindex at a time, but it's fairly
> complicated indexingscript I guess. And of course, the inputfiles
> are huge, mostly beyond 2GB.

scriptindex only pulls a line at a time from the input files, so that
won't matter per se. You may be running into issues with Xapian not
flushing to disk enough - that again 

> topic_title : unhtml weight=10 field=title index=Z index

What is the index=Z intended to do?

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                
  xapian.org
  james at tartarus.org                              
 uncertaintydivision.org

_______________________________________________
Xapian-discuss mailing list
Xapian-discuss at lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss





      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs