Bloquear acesso especifico aos sites que estão no apache2

regislieggio
(usa Lubuntu)

Enviado em 30/09/2014 - 11:52h

Pessoal,

Seguinte eu tenho um apache que será migrado em breve, mas até la ele tem que continuar funcionando.
O pessoal reclama que de madrugada o servidor cai e quando fui ver os logs existe umas requisições do baidu como esta abaixo:

183.60.48.25 - - [06/Jan/2014:15:38:31 -0200] "GET http://www.baidu.com/ HTTP/1.1" 200 13941
123.151.149.222 - - [17/Feb/2014:01:23:32 -0200] "GET http://www.baidu.com/ HTTP/1.1" 200 13941
183.60.48.25 - - [20/Apr/2014:20:56:26 -0300] "GET http://www.baidu.com/ HTTP/1.1" 200 13941
etc...

Não sei se estou errado ou não, mas imagino que seja esse cara que esteja derrubando o servidor.

Alguem saberia me ajudar como bloquear isso?

fabio
(usa Debian)

Enviado em 30/09/2014 - 20:45h

Essa dica é genérica, vai bloquear tanto o Baidu como trocentos outros bots inúteis. Se quiser filtre a lista abaixo e deixe somente as referentes ao Baidu.

Primeiro declare uma lista negra de bots. Pra manter tudo mais organizado, crie a lista num arquivo separado:

# vim /etc/apache2/badbots.conf

# lets create some SetEnvIfNoCase environment cases

<IfModule mod_setenvif.c>



    # Block Bad Bots by User-Agent, doublecheck what you want to be allowed

    SetEnvIfNoCase User-Agent "http://www.baidu.com/search/spider.html" badbotlist



    SetEnvIfNoCase User-Agent "^AESOP_com_SpiderMan" badbotlist

    SetEnvIfNoCase User-Agent "^Alexibot" badbotlist

    SetEnvIfNoCase User-Agent "^asterias" badbotlist

    SetEnvIfNoCase User-Agent "^attach" badbotlist

    SetEnvIfNoCase User-Agent "^BackDoorBot" badbotlist

    SetEnvIfNoCase User-Agent "^BackWeb" badbotlist

    SetEnvIfNoCase User-Agent "Bandit" badbotlist

    SetEnvIfNoCase User-Agent "^Baiduspider" badbotlist

    SetEnvIfNoCase user-Agent "^Baiduspider/2.0" badbotlist

    SetEnvIfNoCase User-Agent "^BatchFTP" badbotlist

    SetEnvIfNoCase User-Agent "^Bigfoot" badbotlist

    SetEnvIfNoCase User-Agent "^Black.Hole" badbotlist

    SetEnvIfNoCase User-Agent "^BlackWidow" badbotlist

    SetEnvIfNoCase User-Agent "^BlowFish" badbotlist

    SetEnvIfNoCase User-Agent "^Bot\ mailto:craftbot@yahoo.com" badbotlist

    SetEnvIfNoCase User-Agent "^BotALot" badbotlist

    SetEnvIfNoCase User-Agent "Buddy" badbotlist

    SetEnvIfNoCase User-Agent "^BuiltBotTough" badbotlist

    SetEnvIfNoCase User-Agent "^Bullseye" badbotlist

    SetEnvIfNoCase User-Agent "^BunnySlippers" badbotlist

    SetEnvIfNoCase User-Agent "^Cegbfeieh" badbotlist

    SetEnvIfNoCase User-Agent "^CheeseBot" badbotlist

    SetEnvIfNoCase User-Agent "^CherryPicker" badbotlist

    SetEnvIfNoCase User-Agent "^ChinaClaw" badbotlist

    SetEnvIfNoCase User-Agent "Collector" badbotlist

    SetEnvIfNoCase User-Agent "Copier" badbotlist

    SetEnvIfNoCase User-Agent "^CopyRightCheck" badbotlist

    SetEnvIfNoCase User-Agent "^cosmos" badbotlist

    SetEnvIfNoCase User-Agent "^Crescent" badbotlist

    SetEnvIfNoCase User-Agent "^Curl" badbotlist

    SetEnvIfNoCase User-Agent "^Custo" badbotlist

    SetEnvIfNoCase User-Agent "^DA" badbotlist

    SetEnvIfNoCase User-Agent "^DISCo" badbotlist

    SetEnvIfNoCase User-Agent "^DIIbot" badbotlist

    SetEnvIfNoCase User-Agent "^DittoSpyder" badbotlist

    SetEnvIfNoCase User-Agent "^Download" badbotlist

    SetEnvIfNoCase User-Agent "^Download\ Demon" badbotlist

    SetEnvIfNoCase User-Agent "^Download\ Devil" badbotlist

    SetEnvIfNoCase User-Agent "^Download\ Wonder" badbotlist

    SetEnvIfNoCase User-Agent "Downloader" badbotlist

    SetEnvIfNoCase User-Agent "^dragonfly" badbotlist

    SetEnvIfNoCase User-Agent "^Drip" badbotlist

    SetEnvIfNoCase User-Agent "^eCatch" badbotlist

    SetEnvIfNoCase User-Agent "^EasyDL" badbotlist

    SetEnvIfNoCase User-Agent "^ebingbong" badbotlist

    SetEnvIfNoCase User-Agent "^EirGrabber" badbotlist

    SetEnvIfNoCase User-Agent "^EmailCollector" badbotlist

    SetEnvIfNoCase User-Agent "^EmailSiphon" badbotlist

    SetEnvIfNoCase User-Agent "^EmailWolf" badbotlist

    SetEnvIfNoCase User-Agent "^EroCrawler" badbotlist

    SetEnvIfNoCase User-Agent "^Exabot" badbotlist

    SetEnvIfNoCase User-Agent "^Express\ WebPictures" badbotlist

    SetEnvIfNoCase User-Agent "Extractor" badbotlist

    SetEnvIfNoCase User-Agent "^EyeNetIE" badbotlist

    SetEnvIfNoCase user-Agent "^Ezooms/1.0" badbotlist

    SetEnvIfNoCase User-Agent "^FileHound" badbotlist

    SetEnvIfNoCase User-Agent "^FlashGet" badbotlist

    SetEnvIfNoCase User-Agent "^Foobot" badbotlist

    SetEnvIfNoCase User-Agent "^flunky" badbotlist

    SetEnvIfNoCase User-Agent "^FrontPage" badbotlist

    SetEnvIfNoCase User-Agent "^GetRight" badbotlist

    SetEnvIfNoCase User-Agent "^GetSmart" badbotlist

    SetEnvIfNoCase User-Agent "^GetWeb!" badbotlist

    SetEnvIfNoCase User-Agent "^Go!Zilla" badbotlist

    SetEnvIfNoCase User-Agent "Google\ Wireless\ Transcoder" badbotlist

    SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" badbotlist

    SetEnvIfNoCase User-Agent "^gotit" badbotlist

    SetEnvIfNoCase User-Agent "Grabber" badbotlist

    SetEnvIfNoCase User-Agent "^GrabNet" badbotlist

    SetEnvIfNoCase User-Agent "^Grafula" badbotlist

    SetEnvIfNoCase User-Agent "^Harvest" badbotlist

    SetEnvIfNoCase User-Agent "^hloader" badbotlist

    SetEnvIfNoCase User-Agent "^HMView" badbotlist

    SetEnvIfNoCase User-Agent "^httplib" badbotlist

    # dont block httrack if you share documentations

    # SetEnvIfNoCase User-Agent "^HTTrack" badbotlist

    SetEnvIfNoCase User-Agent "^humanlinks" badbotlist

    SetEnvIfNoCase User-Agent "^ia_archiver" badbotlist

    SetEnvIfNoCase User-Agent "^IlseBot" badbotlist

    SetEnvIfNoCase User-Agent "^Image\ Stripper" badbotlist

    SetEnvIfNoCase User-Agent "^Image\ Sucker" badbotlist

    SetEnvIfNoCase User-Agent "Indy\ Library" badbotlist

    SetEnvIfNoCase User-Agent "^InfoNaviRobot" badbotlist

    SetEnvIfNoCase User-Agent "^InfoTekies" badbotlist

    SetEnvIfNoCase User-Agent "^Intelliseek" badbotlist

    SetEnvIfNoCase User-Agent "^InterGET" badbotlist

    SetEnvIfNoCase User-Agent "^Internet\ Ninja" badbotlist

    SetEnvIfNoCase User-Agent "^Iria" badbotlist

    SetEnvIfNoCase User-Agent "^Jakarta" badbotlist

    SetEnvIfNoCase User-Agent "^JennyBot" badbotlist

    SetEnvIfNoCase User-Agent "^JetCar" badbotlist

    SetEnvIfNoCase User-Agent "^JOC" badbotlist

    SetEnvIfNoCase User-Agent "^JustView" badbotlist

    SetEnvIfNoCase User-Agent "^Jyxobot" badbotlist

    SetEnvIfNoCase User-Agent "^Kenjin.Spider" badbotlist

    SetEnvIfNoCase User-Agent "^Keyword.Density" badbotlist

    SetEnvIfNoCase User-Agent "^larbin" badbotlist

    SetEnvIfNoCase User-Agent "^LexiBot" badbotlist

    SetEnvIfNoCase User-Agent "^lftp" badbotlist

    SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" badbotlist

    SetEnvIfNoCase User-Agent "^likse" badbotlist

    SetEnvIfNoCase User-Agent "^LinkextractorPro" badbotlist

    SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" badbotlist

    SetEnvIfNoCase User-Agent "^LNSpiderguy" badbotlist

    SetEnvIfNoCase User-Agent "^LinkWalker" badbotlist

    SetEnvIfNoCase User-Agent "^lwp-trivial" badbotlist

    SetEnvIfNoCase User-Agent "^LWP::Simple" badbotlist

    SetEnvIfNoCase User-Agent "^Magnet" badbotlist

    SetEnvIfNoCase User-Agent "^Mag-Net" badbotlist

    SetEnvIfNoCase User-Agent "^MarkWatch" badbotlist

    SetEnvIfNoCase User-Agent "^Mass\ Downloader" badbotlist

    SetEnvIfNoCase User-Agent "^Mata.Hari" badbotlist

    SetEnvIfNoCase User-Agent "^Memo" badbotlist

    SetEnvIfNoCase User-Agent "^Microsoft.URL" badbotlist

    SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" badbotlist

    SetEnvIfNoCase User-Agent "^MIDown\ tool" badbotlist

    SetEnvIfNoCase User-Agent "^MIIxpc" badbotlist

    SetEnvIfNoCase User-Agent "^Mirror" badbotlist

    SetEnvIfNoCase User-Agent "^Missigua\ Locator" badbotlist

    SetEnvIfNoCase User-Agent "^Mister\ PiX" badbotlist

    SetEnvIfNoCase User-Agent "^moget" badbotlist

    SetEnvIfNoCase User-Agent "^NAMEPROTECT" badbotlist

    SetEnvIfNoCase User-Agent "^Navroad" badbotlist

    SetEnvIfNoCase User-Agent "^NearSite" badbotlist

    SetEnvIfNoCase User-Agent "^NetAnts" badbotlist

    SetEnvIfNoCase User-Agent "^NetMechanic" badbotlist

    SetEnvIfNoCase User-Agent "^NetSpider" badbotlist

    SetEnvIfNoCase User-Agent "^Net\ Vampire" badbotlist

    SetEnvIfNoCase User-Agent "^NetZIP" badbotlist

    SetEnvIfNoCase User-Agent "^NextGenSearchBot" badbotlist

    SetEnvIfNoCase User-Agent "^NG" badbotlist

    SetEnvIfNoCase User-Agent "^NICErsPRO" badbotlist

    SetEnvIfNoCase User-Agent "^NimbleCrawler" badbotlist

    SetEnvIfNoCase User-Agent "^Ninja" badbotlist

    SetEnvIfNoCase User-Agent "^NPbot" badbotlist

    SetEnvIfNoCase User-Agent "^Octopus" badbotlist

    SetEnvIfNoCase User-Agent "^Offline\ Explorer" badbotlist

    SetEnvIfNoCase User-Agent "^Offline\ Navigator" badbotlist

    SetEnvIfNoCase User-Agent "^Openfind" badbotlist

    SetEnvIfNoCase User-Agent "^OutfoxBot" badbotlist

    SetEnvIfNoCase User-Agent "^PageGrabber" badbotlist

    SetEnvIfNoCase User-Agent "^Papa\ Foto" badbotlist

    SetEnvIfNoCase User-Agent "^pavuk" badbotlist

    SetEnvIfNoCase User-Agent "^pcBrowser" badbotlist

    SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" badbotlist

    SetEnvIfNoCase User-Agent "^Pockey" badbotlist

    SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" badbotlist

    SetEnvIfNoCase User-Agent "^ProWebWalker" badbotlist

    SetEnvIfNoCase User-Agent "^psbot" badbotlist

    SetEnvIfNoCase User-Agent "^Pump" badbotlist

    SetEnvIfNoCase User-Agent "^QueryN.Metasearch" badbotlist

    SetEnvIfNoCase User-Agent "^RealDownload" badbotlist

    SetEnvIfNoCase User-Agent "Reaper" badbotlist

    SetEnvIfNoCase User-Agent "Recorder" badbotlist

    SetEnvIfNoCase User-Agent "^ReGet" badbotlist

    SetEnvIfNoCase User-Agent "^RepoMonkey" badbotlist

    SetEnvIfNoCase User-Agent "^RMA" badbotlist

    SetEnvIfNoCase User-Agent "Siphon" badbotlist

    SetEnvIfNoCase User-Agent "sitecheck.internetseer.com" badbotlist

    SetEnvIfNoCase User-Agent "^SiteSnagger" badbotlist

    SetEnvIfNoCase User-Agent "^SlySearch" badbotlist

    SetEnvIfNoCase User-Agent "^SmartDownload" badbotlist

    SetEnvIfNoCase User-Agent "^Snake" badbotlist

    SetEnvIfNoCase User-Agent "^Snapbot" badbotlist

    SetEnvIfNoCase User-Agent "^Snoopy" badbotlist

    SetEnvIfNoCase User-Agent "^sogou" badbotlist

    SetEnvIfNoCase User-Agent "^SpaceBison" badbotlist

    SetEnvIfNoCase User-Agent "^SpankBot" badbotlist

    SetEnvIfNoCase User-Agent "^spanner" badbotlist

    SetEnvIfNoCase User-Agent "^Sqworm" badbotlist

    SetEnvIfNoCase User-Agent "Stripper" badbotlist

    SetEnvIfNoCase User-Agent "Sucker" badbotlist

    SetEnvIfNoCase User-Agent "^SuperBot" badbotlist

    SetEnvIfNoCase User-Agent "^SuperHTTP" badbotlist

    SetEnvIfNoCase User-Agent "^Surfbot" badbotlist

    SetEnvIfNoCase User-Agent "^suzuran" badbotlist

    SetEnvIfNoCase User-Agent "^Szukacz/1.4" badbotlist

    SetEnvIfNoCase User-Agent "^tAkeOut" badbotlist

    SetEnvIfNoCase User-Agent "^Teleport" badbotlist

    SetEnvIfNoCase User-Agent "^Telesoft" badbotlist

    SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" badbotlist

    SetEnvIfNoCase User-Agent "^The.Intraformant" badbotlist

    SetEnvIfNoCase User-Agent "^TheNomad" badbotlist

    SetEnvIfNoCase User-Agent "^TightTwatBot" badbotlist

    SetEnvIfNoCase User-Agent "^Titan" badbotlist

    SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" badbotlist

    SetEnvIfNoCase User-Agent "^True_Robot" badbotlist

    SetEnvIfNoCase User-Agent "^turingos" badbotlist

    SetEnvIfNoCase User-Agent "^TurnitinBot" badbotlist

    SetEnvIfNoCase User-Agent "^URLy.Warning" badbotlist

    SetEnvIfNoCase User-Agent "^Vacuum" badbotlist

    SetEnvIfNoCase User-Agent "^VCI" badbotlist

    SetEnvIfNoCase User-Agent "^VoidEYE" badbotlist

    SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" badbotlist

    SetEnvIfNoCase User-Agent "^Web\ Sucker" badbotlist

    SetEnvIfNoCase User-Agent "^WebAuto" badbotlist

    SetEnvIfNoCase User-Agent "^WebBandit" badbotlist

    SetEnvIfNoCase User-Agent "^Webclipping.com" badbotlist

    SetEnvIfNoCase User-Agent "^WebCopier" badbotlist

    SetEnvIfNoCase User-Agent "^webcollage" badbotlist

    SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" badbotlist

    SetEnvIfNoCase User-Agent "^WebEnhancer" badbotlist

    SetEnvIfNoCase User-Agent "^WebFetch" badbotlist

    SetEnvIfNoCase User-Agent "^WebGo\ IS" badbotlist

    SetEnvIfNoCase User-Agent "^Web.Image.Collector" badbotlist

    SetEnvIfNoCase User-Agent "^WebLeacher" badbotlist

    SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" badbotlist

    SetEnvIfNoCase User-Agent "^WebReaper" badbotlist

    SetEnvIfNoCase User-Agent "^WebSauger" badbotlist

    SetEnvIfNoCase User-Agent "^WebSite" badbotlist

    SetEnvIfNoCase User-Agent "^Website\ eXtractor" badbotlist

    SetEnvIfNoCase User-Agent "^Website\ Quester" badbotlist

    SetEnvIfNoCase User-Agent "^Webster" badbotlist

    SetEnvIfNoCase User-Agent "^WebStripper" badbotlist

    SetEnvIfNoCase User-Agent "^WebWhacker" badbotlist

    SetEnvIfNoCase User-Agent "^WebZIP" badbotlist

    SetEnvIfNoCase User-Agent "^Wget" badbotlist

    SetEnvIfNoCase User-Agent "Whacker" badbotlist

    SetEnvIfNoCase User-Agent "^Widow" badbotlist

    SetEnvIfNoCase User-Agent "^WISENutbot" badbotlist

    SetEnvIfNoCase User-Agent "^WWWOFFLE" badbotlist

    SetEnvIfNoCase User-Agent "^WWW-Collector-E" badbotlist

    SetEnvIfNoCase User-Agent "^Xaldon" badbotlist

    SetEnvIfNoCase User-Agent "^Xenu" badbotlist

    SetEnvIfNoCase user-Agent "YandexBot" badbotlist

    SetEnvIfNoCase User-Agent "^Zeus" badbotlist

    SetEnvIfNoCase User-Agent "^Zyborg" badbotlist

    SetEnvIfNoCase User-Agent "ZmEu" badbotlist

<IfModule mod_setenvif.c>

 

Depois acrescente o seguinte no arquivo de configuração do Apache:

# vim /etc/apache2/apache2.conf

Include /etc/apache2/badbots.conf 

E por fim, nas configurações de virtualhosts dos sites que você deseja proteger, adicione o "Deny from env=badbotlist". Exemplo:

<Directory /var/www/site1>

    <Files *>

        Order allow,deny

        Allow from all

        Deny from env=badbotlist

    </Files>

</Directory>

 

Agora é só reiniciar o Apache e boa!

fabio
(usa Debian)

Enviado em 30/09/2014 - 20:47h

Outra solução é criar o arquivo robots.txt no diretório raiz de cada site com o seguinte conteúdo:

User-agent: Baiduspider

User-agent: Baiduspider-video

User-agent: Baiduspider-image

Disallow: /