Re: Search engine robot

Hi Valerie

To make sure the robots skip a page, or specific pages, there are basically
2 ways to go about it:

1. the Meta Tag ->  < META NAME ="robots"    CONTENT ="no index,nofollow">
This will make sure that robots do not index this page, and will not follow
the links on it to other pages. If you want the other pages indexed, then
leave out the "no follow" bit. This goes in between the HEAD tags with the
other META tags. This is good for individual pages, but if you want to
manage it for the site, it is probably easier to just keep a robots.txt file
in your root directory, i.e.,

2. Robots will first look for a file in your directory telling them what's
okay and what's not, so the best thing to do is insert a file in your root
directory for the robots to find, called robots.txt

To exclude all robots from the entire server
User-agent: *       Disallow: /
[This * means every robot - don't use an * anywhere else in the file, such
as *.gif]

To allow all robots complete access
User-agent: *
Disallow:

To exclude a single robot from everything
User-agent: ShagBot
Disallow: /

To allow a single robot everything
User-agent: FemmeBot
Disallow:

To exclude all robots from part of the site:
User-agent: *
Disallow: /private/
[or whatever you want to call the directory.]
It's best to just put everything off limits in one folder, and the pages
that are okay above it.

You can also specify just particular pages, e.g.:
Disallow: /private/grocerylist.html

So, basically, it assumes everything is okay unless you specifically say no.
Kind of like a guy I knew in high school...

HTH

Maggie.



-----Original Message-----
From: williams(at)eburg.com <williams(at)eburg.com>
To: hwg-techniques(at)hwg.org <hwg-techniques(at)hwg.org>
Date: September 18, 2000 6:12 PM
Subject: Search engine robot


:Hi all,
:
:I have a site with javascript password protection and I also want to insert
:the extra code which will cause search engines to ignore the page.  It is
:something like <robot.txt>....I can't recall the details.
:
:Could someone please give me the exact code and the placement.  Is it
:inserted within the head tag like meta tags?
:
:
:Thanks in advance.
:
:
:Valerie Williams
:Websong Productions
:http://www.vocalsplendor.com/websong.html
:
:
:

HWG hwg-techniques mailing list archives, maintained by Webmasters @ IWA