Re: Disallowing files for robots

At 05:34 PM 11/26/00 , Susan Vollmer wrote:
>Does someone have a sample text file that you could
>share or post a link in regard to disallowing robots
>to index specific html pages?
>
>I'm working on a future site that doesn't have hosting
>yet, but I need to temporarily post it on my own site
>for testing, critiques and feedback.

Search engine spiders will NOT know that section is there unless you tell 
them it is by having links on your site (or on another web site) to that 
section / those pages. I do it all the time for clients (and give the 
subdirectories strange names, that helps *g*), and no spider will find the 
pages :-)

>When you create
>a robot.txt file can you disallow for an entire
>directory or must each individual html file be listed?

It's best to do a whole directory. But, as mentioned, that doesn't mean the 
SE spider will obey your request. I block the image spiders from my site. 
Most are pretty good, but there are some sites that would not stop directly 
linking to my images, even after I wrote to them and asked them why they 
were not obeying the robots.txt file. Their attitude was that "it was on 
the 'net, so it's public domain".

Granted, I do have images available for personal use, but when pulled up in 
the image SEs (that would directly link to my server by-the-way!), there is 
no notice of what one can, or can not do, with the images I make available. 
People won't take the time to investigate further, and would use "free for 
personal use" images on their business website :-(

If you want to see another example of a robots.txt file, you can at
http://autumnweb.com/robots.txt

I hope that helps,
Roxanne
Not just putting your business on the Web
Promoting your business on the Web!
Autumn Web ~  http://autumnweb.com/
design / development / promotion / search engine optimization
+ tutorials, web page help, free graphics for personal sites.
* -- * -- * -- * -- * -- * -- * -- * -- * -- * -- *

HWG hwg-techniques mailing list archives, maintained by Webmasters @ IWA