Fail2ban

fail2ban is a tool designed to block suspicious webserver visits:

Apache log files are scanned in real-time for suspicious behaviour. Both the log files and the filters are customizable. This is done through Python
Selected lines are added to iptables' chains - So, the actual blocking is done through iptables, which seems like a very efficient approach to blocking stuff
Duration of these blocking rules is configurable. E.g., an hour for repeated inlog failures, or a week for what seems to be a hacked computer in a botnet

Existing rules flushed on restart

By default, when fail2ban is restarted, it flushes the existing iptables rules that it manages. This happens because fail2ban is designed to reinitialize its own firewall rules upon starting or restarting. Specifically, it clears and rebuilds the rules in the chains it manages (typically named f2b-*), which means any existing bans applied by fail2ban will be removed and need to be re-added as fail2ban reads the log files and re-applies bans.

Note that fail2ban probably doesn't flush rules outside the f2b-* chains: Those might get flushed when restarting a webserver, as iptables' rules are not persistent across restarts of iptables.

Keep the whitelist on top

I have a set of rules, or chain, called whitelist. This chain or rules set, should always be on top of the rules (before fail2ban chains), also when fail2ban, iptables or the server get restarted. Some solutions:

Mark the whilelist chain as independent: Just make shure it's name doesn't start with f2b-: Fail2ban supposingly won't flush it, but it's still gone when iptables or the server gets restarted
postrouting: Add postrouting hook at the start of the INPUT chain, that the whitelist chain should be executed first - Doesn't solve the iptables and/or server restart issue
Automatically load whitelist on server start: Quite obvious solution. Only doesn't work when iptables is restarted, but when does that ever happen?

Introduction

What it does

Monitors Log Files: Fail2ban scans log files for specific patterns that indicate suspicious activity, such as failed login attempts or other anomalies
Blocks IP Addresses: When it detects repeated failed attempts from a particular IP address, it can temporarily or permanently block that IP address using firewall rules (such as with iptables, nftables, or firewalld)
Configurable: You can customize Fail2ban to monitor different services and define what constitutes "bad behavior." It can protect various services like SSH, FTP, HTTP, and more.

How it works

Jails: Fail2ban uses "jails," which are configurations specifying which log files to monitor, what patterns to look for, and what actions to take. Each jail is associated with a particular service or application
Filters: Filters are used within jails to define the patterns of failed attempts. These are often specified using regular expressions
Actions: When a pattern is detected, Fail2ban can execute various actions, typically involving updating firewall rules to block the offending IP addresses.

Configuration

Main Configuration File: /etc/fail2ban/jail.conf (though it's a good practice to create and modify /etc/fail2ban/jail.local to avoid overwriting the default settings)
Filters Directory: /etc/fail2ban/filter.d/ contains filter definitions
Actions Directory: /etc/fail2ban/action.d/ contains predefined actions for blocking IPs.

Benefits

Flexible: Fail2ban can probably be configured for any network function that maintains a log file
Makes WordPress security plugins superfluous? Would be nice if I can dump WordFence
Efficient: Fail2ban has been written in Python, while the actual banning is (in our case) done by iptables. Using a low-level tool like iptables for filtering, is much more efficient than using WordPress plugins like WordFence
Temporary blocking: It's really nice that blocks can be temporary, so in case someone genuinely makes 20 mistakes in a row, or if a hacked IP address eventually gets unhacked, that they regain access.

Status

Example of an active instance:

$ sudo systemctl status fail2ban

● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-09-18 04:05:35 CEST; 3 weeks 2 days ago
       Docs: man:fail2ban(1)
   Main PID: 2350 (fail2ban-server)
      Tasks: 15 (limit: 115387)
     Memory: 30.4M (peak: 49.2M swap: 2.3M swap peak: 2.3M)
        CPU: 2h 48min 56.177s
     CGroup: /system.slice/fail2ban.service
             └─2350 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

Notice: journal has been rotated since unit was started, output may be incomplete.

Example of a paused instance:

$ sudo systemctl status fail2ban

○ fail2ban.service - Fail2Ban Service
     Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; preset: enabled)
     Active: inactive (dead) since Fri 2024-10-11 12:41:25 CEST; 54s ago
   Duration: 3w 2d 8h 35min 48.694s
       Docs: man:fail2ban(1)
    Process: 2350 ExecStart=/usr/bin/fail2ban-server -xf start (code=killed, signal=TERM)
    Process: 2407962 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=0/SUCCESS)
   Main PID: 2350 (code=killed, signal=TERM)
        CPU: 2h 48min 56.608s

Oct 11 12:41:24 server2 systemd[1]: Stopping fail2ban.service - Fail2Ban Service...
Oct 11 12:41:25 server2 fail2ban-client[2407962]: Shutdown successful
Oct 11 12:41:25 server2 systemd[1]: fail2ban.service: Deactivated successfully.
Oct 11 12:41:25 server2 systemd[1]: Stopped fail2ban.service - Fail2Ban Service.
Oct 11 12:41:25 server2 systemd[1]: fail2ban.service: Consumed 2h 48min 56.608s CPU time, 49.2M memory peak, 2.3M memory >
Notice: journal has been rotated since unit was started, output may be incomplete.

Start & stop

Temporarily stop Fail2Ban:

sudo systemctl stop fail2ban

Permanently disable Fail2Ban:

sudo systemctl disable fail2ban

Re-enable:

sudo systemctl enable fail2ban

Restart:

sudo systemctl stop fail2ban

Sources

https://en.wikipedia.org/wiki/Fail2ban
https://github.com/fail2ban/fail2ban - Home page