Re: Search engine robot
by "Maggie Carr" <m.a.carr(at)decaelo.com>
||Mon, 18 Sep 2000 20:33:01 -0400
To make sure the robots skip a page, or specific pages, there are basically
2 ways to go about it:
1. the Meta Tag -> < META NAME ="robots" CONTENT ="no index,nofollow">
This will make sure that robots do not index this page, and will not follow
the links on it to other pages. If you want the other pages indexed, then
leave out the "no follow" bit. This goes in between the HEAD tags with the
other META tags. This is good for individual pages, but if you want to
manage it for the site, it is probably easier to just keep a robots.txt file
in your root directory, i.e.,
2. Robots will first look for a file in your directory telling them what's
okay and what's not, so the best thing to do is insert a file in your root
directory for the robots to find, called robots.txt
To exclude all robots from the entire server
User-agent: * Disallow: /
[This * means every robot - don't use an * anywhere else in the file, such
To allow all robots complete access
To exclude a single robot from everything
To allow a single robot everything
To exclude all robots from part of the site:
[or whatever you want to call the directory.]
It's best to just put everything off limits in one folder, and the pages
that are okay above it.
You can also specify just particular pages, e.g.:
So, basically, it assumes everything is okay unless you specifically say no.
Kind of like a guy I knew in high school...
From: williams(at)eburg.com <williams(at)eburg.com>
To: hwg-techniques(at)hwg.org <hwg-techniques(at)hwg.org>
Date: September 18, 2000 6:12 PM
Subject: Search engine robot
:the extra code which will cause search engines to ignore the page. It is
:something like <robot.txt>....I can't recall the details.
:Could someone please give me the exact code and the placement. Is it
:inserted within the head tag like meta tags?
:Thanks in advance.
HWG hwg-techniques mailing list archives,
maintained by Webmasters @ IWA