[Xapian-discuss] How to update DB concurrently?

[oscaruser at programmer.net]

seems xapian-tcpsrv is what was needed, while increasing listen backlog to 256 in tcpserver.cc

thanks

> ----- Original Message -----
> From: oscaruser at programmer.net
> To: xapian-discuss at lists.xapian.org
> Subject: [Xapian-discuss] How to update DB concurrently?
> Date: Wed, 17 May 2006 08:59:01 -0800
> 
> 
> Folks,
> 
> I have many (~150) perl spider processes that gathers data from 
> targeted URLs running concurrently. They use mysql to maintain 
> state, and locking is handled only if two spider processes happen 
> to randomly select the same URL to process. I want to use omindex 
> to add the data to the index being built from the data source HTML 
> downloaded near the start of the spidering process. However, each 
> omindex process creates a db_lock file before modifying the index 
> data, so other processes are locked out. Also, it would be nice to 
> be able to search the index while it is updated and grows. It seems 
> the db_lock does not prevent searching. What is a better way to do 
> this than waiting for omindex to exit? One thought is to modify 
> omindex to create a server that accepts IPC named pipe or socket 
> connections. The spiders would connect and send data over then 
> close the connection. Has this been done anywhere so that I need 
> not write it?
> 
> Thanks,
> OSC
> 
> --
> ___________________________________________________
> Play 100s of games for FREE! http://games.mail.com/
> 
> 
> _______________________________________________
> Xapian-discuss mailing list
> Xapian-discuss at lists.xapian.org
> http://lists.xapian.org/mailman/listinfo/xapian-discuss

>


-- 
___________________________________________________
Play 100s of games for FREE! http://games.mail.com/