Re: Custom pages for robots!

At 11:00 AM 12/2/99 , Rich Bowen wrote:

>Apache could certainly do that without much difficulty, using SetEnv or
>something like that. I suppose that one could even go so far a to write
>a handler to do something like that. I've not heard of a stand-alone
>product to do that. But a clever arrangement of SetEnvIf, BrowserMatch,
>and perhaps some other magic to look up a list of robots, could do this.

Rich,

Thanks for the reply, .. I would think that doing something like that as an individual developer would not be worth the effort - it would take a *considerable* amount of time to try and reverse engineer the major search engines, recognizing their robots, and customizing specific approaches.

>It's a shame that people feel that they have to cheat to get ahead of
>the system. I fail to see how a dishonestly indexed site is beneficial
>to the user.

I don't think that cheating is the issue at all. The issue is that the search engines all look for *different* things when indexing (as I understand it, .. but I am not an expert <g>, .. the reason for this post). Especially when considering any sort of dynamic or frames-based site. Is that the fault of the site developer?

Creating an index page formatted specifically for a search engine does nothing more than format the information correctly for THAT search engine. A completely different concept than the more 'fraudalent' applications you might be thinking about, e.g. embedding Christian keywords into a porno site.

         Lee
============================================
    Leland V. Lammert                                lvl(at)omnitec.net
       Chief Scientist                         Omnitec Corporation
   Network/Internet Consultants              www.omnitec.net
============================================

HWG: hwg-servers mailing list archives, maintained by Webmasters @ IWA