Hello Rachel,

I can guide you through some steps to exclude bot agents using ModSecurityAnd I assume you have already installed it. If you haven't, then follow me.

We already have an included directory for ModSecurity rules in the Apache configuration setup.

Configure ModSecurity to Block bot Agents

Include /etc/httpd/conf/modsecurity.d/rules/*.conf

Now, we need to create a new .conf to start detecting agents.

nano /etc/httpd/conf/modsecurity.d/rules/block_user_agents.conf

We will add the following and save the file.

SecRule REQUEST_HEADERS:User-Agent "@pmFromFile badbots.txt" "id:350001,rev:1,severity:2,log,msg:'BAD BOT - Detected and Blocked. '"

We will create the list of User Agents to be detected and blocked:

nano /etc/httpd/conf/modsecurity.d/rules/badbots.txt
Then you can insert the list of the user agents if you want to allow them. like,

Baidu
User-Agent

Now, We need to configure Fail2ban

First, you will need to install Fail2ban

yum -y install fail2ban

Now you need to create a new jail file,

nano /etc/fail2ban/jail.local

local jail allows the main fail2ban configuration to be updated with the new updates

[apache-modsecblock-badbots]
enabled = true
filter = apache-useragent
logpath = /var/log/httpd/error_log
action = iptables-multiport[name=apache-badbots, port="http,https", protocol=tcp]
postback[name=BADBOT, port="http,https", protocol=tcp]
maxretry = 2
bantime = 172800
ignoreip = 127.0.0.0/8 10.0.0.0/8 192.168.1.0/24

Update ignoreip with any local IPs or any others you want to allow in regardless of the UserAgent. This allows each IP to access twice with the UserAgent indicated in the list, after that it will be banned.

You will then want to create the failregex pattern

nano /etc/fail2ban/filter.d/apache-useragent.conf

And you need to add the following:

# Fail2Ban configuration file
# [Definition] # Option: failregex
# Notes.: Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# abovementioned bots.
# Values: TEXT
# failregex = [[]client <HOST>[]] ModSecurity: Access denied with code 406 .* [[]msg "BAD BOT - Detected and Blocked. "[]] .*$ # Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#

Go ahead and restart fail2ban and apache.

service httpd restart service fail2ban restart

Hope you will find this helpful.