[Xapian-discuss] newbie help, group site crawl, index and search

[Haniff Din]

Hello xapian-discuss,

  Forgive the newbie question.
  
  I want to crawl a subset of business sites on a regular basis, in a specific
  niche market (about 100-200 sites).
  In effect for  parent charity organisation website so you can search all the group
  independent sites, from the parent, as if was one web-site.
  Of course this means, I always crawl and index the same group
  of sites, which is expanding slowly in effect. So the crawler
  is always given the same set of URLs to crawl in effect.
  
  What is the best approach to crawl and build an index
  for searching?  Are crawl and/or index tools part
  of this tool-set? Are they available?

  Appreciate any help, as this is a non profit, charity, it
  will really help people.

-- 
Best regards,
 Haniff 

 mailto:mail at haniff.net

 www.haniff.co.uk