[Xapian-discuss] omega crawler: ht://dig or wget?

[James Aylett]

On Fri, Mar 17, 2006 at 06:33:02AM +0000, Olly Betts wrote:

> > Would be nice if fetcher can log in into some sites with password or cookie.
> > Currently I do not see any more features. Can wget or htdig do it?
> 
> I suspect either can.  If wget seems more suitable for your needs, go
> ahead and use it.  If you want to write a HOWTO to help others, or
> perhaps some sort of wrapper script please do.

wget cookie mechanism:

--header "Cookie: COOKIENAME=value"

wget HTTP auth mechanism:

--http-user=USER --http-passwd=PASS

The latter is trivially in the help/man page, the former requires a
knowledge of HTTP. Or you can use wget's inbuilt cookie handling,
although I've no idea if it's compatible with an easy way of picking
them up (eg: lynx).

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org