How To Block Bad Bots in WHM/cPanel servers.
This blog explains How To Block Bad Bots in WHM/cPanel servers.
The Internet has lots of unwanted traffic, which causes a high load on your dedicated or virtual private server. Traffic can be from bot networks, A Web crawler, or normal web traffic from different sources.
Bot crawlers are essential algorithms that index your website. However, that applies to only a few of them as some bots cause more harm than good, like server load or network instability.
Get assistance from us to manage your cPanel server here.
Here are a few steps To Block Bad Bots in WHM/cPanel servers
1) To Block Bots from Apache with WHM/cPanel
1. Log in to WHM using your root account.
2. Select the Service Configuration option from the navigation menu.
3. Click on Apache Configuration. The list of configuration options for the Apache server will appear.
4. Click on Include Editor. Here, you can apply configurations to your Apache server on a global scale.
5. Go to the Pre Main Include section and, under I wish to edit the Pre Main configuration include file for: select All Versions from the dropdown list. A text box will appear in the Pre Main Include section.
6. Copy/enter the following codes in there and click the Update button.
<directory "="" home"=""> SetEnvIfNoCase User-Agent "MJ12bot" bad_bots SetEnvIfNoCase User-Agent "coccocbot-image" bad_bots SetEnvIfNoCase User-Agent "Baiduspider" bad_bots SetEnvIfNoCase User-Agent "AhrefsBot" bad_bots SetEnvIfNoCase User-Agent "SemrushBot" bad_bots SetEnvIfNoCase User-Agent "DotBot" bad_bots SetEnvIfNoCase User-Agent "AlphaBot" bad_bots SetEnvIfNoCase User-Agent "ZoominfoBot" bad_bots SetEnvIfNoCase User-Agent "ADmantX" bad_bots SetEnvIfNoCase User-Agent "Heritrix" bad_bots SetEnvIfNoCase User-Agent "Indy Library" bad_bots SetEnvIfNoCase User-Agent "Mail.Ru" bad_bots SetEnvIfNoCase User-Agent "rogerbot" bad_bots SetEnvIfNoCase User-Agent "PHPCrawl" bad_bots SetEnvIfNoCase User-Agent "BLEXBot" bad_bots SetEnvIfNoCase User-Agent "magpie-crawler" bad_bots SetEnvIfNoCase User-Agent "SeznamBot" bad_bots SetEnvIfNoCase User-Agent "seoscanners.net" bad_bots SetEnvIfNoCase User-Agent "ZumBot" bad_bots SetEnvIfNoCase User-Agent "Yandex" bad_bots SetEnvIfNoCase User-Agent "MaxPointCrawler" bad_bots SetEnvIfNoCase User-Agent "Nutch" bad_bots SetEnvIfNoCase User-Agent "Buzzbot" bad_bots Require all granted Require not env bad_bots
2) To Block Bots with robot.txt File
Go to document root for the requested domain and create a file as per the below.
# vi ~username/public_html/robot.txt
User-agent: *
Disallow: /
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule .* – [F,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza|Datanyze|serpstatbot|spaziodati|OPPOsA33|AspiegelBot|aspiegel|PetalBot [NC]
RewriteRule .* – [F,L]
Note: You may wish to edit the list of bots above, especially if your SEO company uses something like AHREFS or SEMRush. If you are seeing a high load from such bots some will respect the robots.txt file – so you can simply set limits for such bots like this via your robots.txt file:
# vi ~username/public_html/robot.txt
User-agent: SemrushBot Crawl-delay: 20 :wq
Note: Don’t forget to update the ownership for that file.
# chown username:username ~username/public_html/robot.txt
3) To Block Bots with the .htaccess File
 If you wanted to block multiple User-Agent strings at once, you could do it like this:
# vi ~username/public_html/.htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(Baiduspider|HTTrack|Yandex).*$ [NC]
RewriteRule .* - [F,L]
:wq
 Or you can also use the BrowserMatchNoCase directive like this:
# vi ~username/public_html/.htaccess
BrowserMatchNoCase "Baiduspider" bots BrowserMatchNoCase "HTTrack" bots BrowserMatchNoCase "Yandex" bots Order Allow,Deny Allow from ALL Deny from env=bots
4) To Block the Bots with Mod Security rule
1. Go to WHM Home » Security Center » ModSecurity™ Tools » Rules List » Add Rule
2. Copy/enter the following rules in there (any one rule which wants to need)
SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:MJ12bot|AhrefsBot)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403" SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:AhrefsBot|MJ12bot|Yandex)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403" SecRule REQUEST_HEADERS:User-Agent "@rx (?:ahrefsbot|mj12bot|yandex)" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:lowercase,block,status:403" SecRule REQUEST_HEADERS:User-Agent "@rx (?:AhrefsBot)" "msg:'AhrefsBot Spiderbot blocked',phase:1,log,id:7777771,t:none,block,status:403" SecRule REQUEST_HEADERS:User-Agent "@rx (?:MJ12bot)" "msg:'MJ12bot Spiderbot blocked',phase:1,log,id:7777772,t:none,block,status:403" SecRule REQUEST_HEADERS:User-Agent "@rx (?:Yandex)" "msg:'Yandex Spiderbot blocked',phase:1,log,id:7777773,t:none,block,status:403"
By using this snippet we can able to Block Bad Bots in WHM/cPanel servers.
