Re: Banning Bad Bots by UA

Thanks Rich, that was very helpful. Would you happen to know if the syntax 
would allow for banning multiple user agents in the form of..

SetEnvIf User-Agent Wget, Emailsiphon, ExtractorPro GoAway

or is an entire set of directives needed for each UA?

I have tried several combinations but it seems to work only with one UA at 
a time which is indeed helpful, but banning a "list" of these critters 
would be most excellent.

Thanks again,

Jim
At 07:07 AM 6/23/01 -0400, Rich Bowen wrote:

>On Sat, 16 Jun 2001 18:55:43 -0500, jim said:
>
> > Hi folks,
> >
> >  I was intending to use .htaccess to ban bots based on User Agent being 
> used
> >  by less than the nicest of Netizens using Spiders and Bots that do not
> >  follow exclusion protocol. I'm running Apache servers but the build is to
> >  support Frontpage 2000 and has problems such that mod_rewrite won't 
> compile
> >  in without going through a lot of pain. Any ideas that won't make me 
> go the
> >  SSL route?
>
>Sorry, I missed this note earlier.
>
>See http://apachetoday.com/news_story.php3?ltsn=2000-11-13-003-01-SC-LF-SW
>which talks about denying access based on a variety of different things. There
>are two things you want to do. You need to explicitly deny by host, if you 
>know
>for sure which hosts are spidering you. And you need to deny by user agent,
>which is discussed later in the article, when I discuss "SetEnvIf".
>
>Hope this helps.
>
>--
>Rich Bowen - rbowen(at)rcbowen.com
>http://www.rcbowen.com/kenya/

HWG: hwg-servers mailing list archives, maintained by Webmasters @ IWA