Assistanz
Assistanz

How To Block Bad Bots in WHM/cPanel servers

How To Block Bad Bots in WHM/cPanel servers

How To Block Bad Bots in WHM/cPanel servers.

This blog explains How To Block Bad Bots in WHM/cPanel servers.

The Internet has lots of unwanted traffic, which causes a high load on your dedicated or virtual private server. Traffic can be from bot networks, A Web crawler, or normal web traffic from different sources.

Bot crawlers are essential algorithms that index your website. However, that applies to only a few of them as some bots cause more harm than good, like server load or network instability.

Get assistance from us to manage your cPanel server here.

 

Here are a few steps To Block Bad Bots in WHM/cPanel servers

 

1) To Block Bots from Apache with WHM/cPanel

 

1. Log in to WHM using your root account.
2. Select the Service Configuration option from the navigation menu.
3. Click on Apache Configuration. The list of configuration options for the Apache server will appear.
4. Click on Include Editor. Here, you can apply configurations to your Apache server on a global scale.
5. Go to the Pre Main Include section and, under I wish to edit the Pre Main configuration include file for: select All Versions from the dropdown list. A text box will appear in the Pre Main Include section.
6. Copy/enter the following codes in there and click the Update button.

 

<directory "="" home"="">
SetEnvIfNoCase User-Agent "MJ12bot" bad_bots
SetEnvIfNoCase User-Agent "coccocbot-image" bad_bots
SetEnvIfNoCase User-Agent "Baiduspider" bad_bots
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bots
SetEnvIfNoCase User-Agent "SemrushBot" bad_bots
SetEnvIfNoCase User-Agent "DotBot" bad_bots
SetEnvIfNoCase User-Agent "AlphaBot" bad_bots
SetEnvIfNoCase User-Agent "ZoominfoBot" bad_bots
SetEnvIfNoCase User-Agent "ADmantX" bad_bots
SetEnvIfNoCase User-Agent "Heritrix" bad_bots
SetEnvIfNoCase User-Agent "Indy Library" bad_bots
SetEnvIfNoCase User-Agent "Mail.Ru" bad_bots
SetEnvIfNoCase User-Agent "rogerbot" bad_bots
SetEnvIfNoCase User-Agent "PHPCrawl" bad_bots
SetEnvIfNoCase User-Agent "BLEXBot" bad_bots
SetEnvIfNoCase User-Agent "magpie-crawler" bad_bots
SetEnvIfNoCase User-Agent "SeznamBot" bad_bots
SetEnvIfNoCase User-Agent "seoscanners.net" bad_bots
SetEnvIfNoCase User-Agent "ZumBot" bad_bots
SetEnvIfNoCase User-Agent "Yandex" bad_bots
SetEnvIfNoCase User-Agent "MaxPointCrawler" bad_bots
SetEnvIfNoCase User-Agent "Nutch" bad_bots
SetEnvIfNoCase User-Agent "Buzzbot" bad_bots

Require all granted
Require not env bad_bots

 

2) To Block Bots with robot.txt File

 

Go to document root for the requested domain and create a file as per the below.

# vi ~username/public_html/robot.txt

User-agent: *
Disallow: /
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule .* – [F,L]
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza|Datanyze|serpstatbot|spaziodati|OPPOsA33|AspiegelBot|aspiegel|PetalBot [NC]
RewriteRule .* – [F,L]

 

Note: You may wish to edit the list of bots above, especially if your SEO company uses something like AHREFS or SEMRush. If you are seeing a high load from such bots some will respect the robots.txt file – so you can simply set limits for such bots like this via your robots.txt file:

# vi ~username/public_html/robot.txt

User-agent: SemrushBot
Crawl-delay: 20
:wq

 

Note: Don’t forget to update the ownership for that file.

# chown username:username ~username/public_html/robot.txt

 

3) To Block Bots with the .htaccess File

 

 If you wanted to block multiple User-Agent strings at once, you could do it like this:

# vi ~username/public_html/.htaccess

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(Baiduspider|HTTrack|Yandex).*$ [NC]
RewriteRule .* - [F,L]

 

:wq

 Or you can also use the BrowserMatchNoCase directive like this:
# vi ~username/public_html/.htaccess

BrowserMatchNoCase "Baiduspider" bots
BrowserMatchNoCase "HTTrack" bots
BrowserMatchNoCase "Yandex" bots
Order Allow,Deny
Allow from ALL
Deny from env=bots

 

4) To Block the Bots with Mod Security rule

 

1. Go to WHM Home » Security Center » ModSecurity™ Tools » Rules List » Add Rule

2. Copy/enter the following rules in there (any one rule which wants to need)

SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:MJ12bot|AhrefsBot)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:AhrefsBot|MJ12bot|Yandex)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:ahrefsbot|mj12bot|yandex)" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:lowercase,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:AhrefsBot)" "msg:'AhrefsBot Spiderbot blocked',phase:1,log,id:7777771,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:MJ12bot)" "msg:'MJ12bot Spiderbot blocked',phase:1,log,id:7777772,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:Yandex)" "msg:'Yandex Spiderbot blocked',phase:1,log,id:7777773,t:none,block,status:403"

 

By using this snippet we can able to Block Bad Bots in WHM/cPanel servers.

 

 

DirectAdmin Server Management Plans

Premium Support

24/7 End User Support from your Helpdesk
$ 99 Monthly / Server
  • Unlimited Support Plan
  • 24/7 Emergency Phone
  • Chat Support for Admin
  • Separate Account Manager
  • NDA & SLA
  • SLA Review Meetings
  • FREE Consultancy Services
  • Simple SignUp Process
  • Instant Account Activation
  •  

Platinum Support

24/7 Proactive Support
$ 49 Monthly / Server
  • Unlimited Support Plan
  • 24/7 Emergency Phone
  • Chat Support for Admin
  • Separate Account Manager
  • Advance Proactive Monitoring
  • Guaranteed SLA
  • SLA Review Meetings
  • 3rd Party Application Support
  • FREE Consultancy Services
  • Server Migration Support
  • Weekly Status Report
  • No End User Support
Popular

Unlimited Support

24/7 Support

$ 30 Monthly / Server
  • Unlimited Support Plan
  • 24/7 Support
  • Basic Monitoring
  • 30 Minutes Response Time
  • 4 Hours Resolution time for the Possible Issues
  • Security and Performance Optimization
  • React to Customers queries
  • Simple SignUp Process
  • Instant Account Activation
  • No Third party application support
  • No Migration and End User Support

Cpanel Server Management Plans

Premium Support

24/7 End User Support from your Helpdesk

$ 99 Monthly / Server
  • 24/7 End User White Label Support
  • Unlimited Number of Tickets
  • Chat Support for Admin
  • Separate Account Manager
  • Guaranteed SLA
  • Weekly Status Reports
  • FREE Consultancy Services
  • Simple SignUp Process
  • Instant Account Activation

Platinum Support

24/7 Proactive Server Management
$ 49 Monthly / Server
  • Unlimited Support Plan
  • 24/7 Emergency Phone
  • Chat Support for Admin
  • Separate Account Manager
  • Advance Proactive Monitoring
  • Guaranteed SLA
  • SLA Review Meetings
  • 3rd Party Application Support
  • FREE Consultancy Services
  • Server Migration Support
  • Weekly Status Report
  • No End User Support
Popular

Unlimited Support

24/7 Server Management

$ 29 Monthly / Server
  • Unlimited Admin Tasks
  • 24/7 Support
  • Basic Monitoring
  • 30 Minutes Response time
  • 4 hours Response time for possible issues
  • Security and Performance Optimization
  • React to Customers queries
  • No Third party application support
  • No Migration and End User Support

Cpanel Server Management Plans

Premium Support

24/7 End User Support from your Helpdesk

$ 99 Monthly / Server
  • 24/7 End User White Label Support
  • Unlimited Number of Tickets
  • Chat Support for Admin
  • Separate Account Manager
  • Guaranteed SLA
  • Weekly Status Reports
  • FREE Consultancy Services
  • Simple SignUp Process
  • Instant Account Activation

Platinum Support

24/7 Proactive Server Management
$ 49 Monthly / Server
  • Unlimited Support Plan
  • 24/7 Emergency Phone
  • Chat Support for Admin
  • Separate Account Manager
  • Advance Proactive Monitoring
  • Guaranteed SLA
  • SLA Review Meetings
  • 3rd Party Application Support
  • FREE Consultancy Services
  • Server Migration Support
  • Weekly Status Report
  • No End User Support
Popular

Unlimited Support

24/7 Server Management

$ 29 Monthly / Server
  • Unlimited Admin Tasks
  • 24/7 Support
  • Basic Monitoring
  • 30 Minutes Response time
  • 4 hours Response time for possible issues
  • Security and Performance Optimization
  • React to Customers queries
  • No Third party application support
  • No Migration and End User Support