Re: Crawling a Site

by Donna Smillie <dms(at)zetnet.co.uk>

Date:	Sun, 14 Sep 2003 16:33:42 +0100
To:	HWG Techniques <hwg-techniques(at)hwg.org>
References:	web fastmail
	todo: View Thread, Original

>beno wrote:
>> Hi;
>> Isn't there some program out there that will crawl a competitor's site=
 for=20
>> me and give me the site map?

Black Widow, by Softbyte Labs (see www.softbytelabs.com), will do this
amongst other things.  From their website:

"BlackWidow is a web site scanner, a site mapping tool,
a site ripper, a site mirroring tool, and an offline browser program."

I've used it a couple of times, and it seems to do quite a good job.

Donna

HWG hwg-techniques mailing list archives, maintained by Webmasters @ IWA