Re: Robots and Cookie problems

by Thomas James Allen <tjallen(at)>

 Date:  Fri, 24 Sep 1999 02:35:01 -0400
 To:  "Rajnish Bhaskar" <9705228b(at)>
 Cc:  hwg-basics(at)
  todo: View Thread, Original
At 07:29 PM 9/23/99 +0100, you wrote:
>Hi all,
>And finally, according to my HTML book, a robots.txt can be used to 
>disallow individual pages from being spidered, but I seem to recall 
>someone on this list saying that robots.txt can only be used to disallow 
>whole directories.  Am I imagining things, or is this actually the case?
>Thanks a lot,
For good info on robots.txt, see:

Includes an extensive FAQ.
Includes info on avoiding being indexed, with a meta tag and robot.txt

Hope this helps,
Jimmy Allen (Thomas James Allen)
Business - Get Your Website Here
Personal - Eclecticity

HTML: hwg-basics mailing list archives, maintained by Webmasters @ IWA