as opposed to doing detection by ip, most robots have a specific user agent,
like the w3c's validator uses

W3C_Validator/1.34 libwww-perl/5.43

if you can figure out what the user agents of the robots it should be easy
enough

a site i was working on used different versions of html depending on the
user agent (like  most people do) but i made it serve up strict html 4 to
the w3c's validator because someone insisted on testing the validity of the
site

the same basic principal can be applied to search engine robots, if you send
them to a dummy page you can see what the user agent is and use that when
you serve up the  page you want them to see

-----Original Message-----
From: Rich Bowen [mailto:rbowen(at)rcbowen.com]
Sent: Thursday, December 02, 1999 12:00 PM
To: Leland V. Lammert
Cc: hwg-servers(at)hwg.org
Subject: Re: Custom pages for robots!


"Leland V. Lammert" wrote:
> 
> I recently learned of a product that is supposed to recognize a robot by
it's IP and then serve a custom page designed to meet the criteria for that
specific search engine to index, while normal users receive the standard
index.html. Supposedly, it makes the process of ranking in a specific search
engine system many times higher than allowing the robot to actually index
the site pages
> 
> Has anyone heard of anything like this?

Apache could certainly do that without much difficulty, using SetEnv or
something like that. I suppose that one could even go so far a to write
a handler to do something like that. I've not heard of a stand-alone
product to do that. But a clever arrangement of SetEnvIf, BrowserMatch,
and perhaps some other magic to look up a list of robots, could do this.

It's a shame that people feel that they have to cheat to get ahead of
the system. I fail to see how a dishonestly indexed site is beneficial
to the user.

Rich
-- 
http://www.ApacheUnleashed.com/
Lexington Perl Mongers - http://lexington.pm.org/
PGP Key - http://www.rcbowen.com/pgp.txt

• Web Designer Courses
• Off the Shelf Training Courses

HWG: hwg-servers mailing list archives, maintained by Webmasters @ IWA