[Xapian-discuss] PHP indexing, what's the PHP method for indexscript

[athlon athlonf]

>Load 5 suggests something's wrong, because dbi2omega and scriptindex
>are both linear processes. Are you running several instances in
>parallel in some way?

it usually starts off fairly low, but then after half an hour of so, it will reach load 5 constantly.
I'm only doing one scriptindex at a time, but it's fairly complicated indexingscript I guess. And of course, the inputfiles are huge, mostly beyond 2GB.


>I believe that right now, none of the supplied Xapian indexing scripts
>or binaries will go significantly above a load of 1, unless you have
>other issues or something else happening on the machine.



The machine is on itself also a webserver, but the normal average load is below 0.25. I can imagine it to get into memory problem when scriptindex tries to parse 2GB worth of data each time.


This is the indexscript i was using.

tid : boolean=Q field=id
pid : unique=Q boolean=Q field=pid
topic_title : unhtml weight=10 field=title index=Z index
forum_id : indexnopos=XFORUMID field=forum_id
postdate : field=startdate
postdateh : field=startdateh  date=yyyymmdd value=2
topicStart : field=topicStart
topicStarth : field=topicStarth  date=yyyymmdd value=3
author_name: unhtml boolean=XAUTHORNAME field=author
author_id: boolean=XAUTHORID field=authorid
state : field=state
url :  field=url
sample : weight=1 index field=sample



      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ