-
-
Notifications
You must be signed in to change notification settings - Fork 82
Description
Support guidelines
- I've read the support guidelines
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
When running fail2ban as a docker container on the same host as my nginx proxy manager docker container the system slows down over a 24 hour period
Expected behaviour
For the system to not slow down even if left alone for a week or more
Actual behaviour
I have two servers that are functionally identical - two Raspberry Pi 4B 2GB boards with 120GB SSD via USB3 connectors. They run edge services using docker compose files.
edge1: Linux edge1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
edge2: Linux edge2 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
edge1 containers
```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-1 2B (virtual 40.3MB) edge1 Up 3 minutes 4 days ago running diun-1 0B (virtual 54.6MB) edge1 Up 3 minutes 4 days ago running fail2ban 3.12kB (virtual 90.9MB) host Up 3 minutes (healthy) 4 days ago running netdata-1 707kB (virtual 395MB) edge1 Up 3 minutes (healthy) 23 hours ago running npm 42.4MB (virtual 858MB) edge1 Up 3 minutes (healthy) 4 days ago running pihole-1 74.2MB (virtual 368MB) host Up 3 minutes (healthy) 23 hours ago running promtail-1 0B (virtual 185MB) edge1 Up 3 minutes 3 days ago running scrutiny-collector-1 9.37kB (virtual 189MB) edge1 Up 3 minutes 4 days ago running wireguard-1 23kB (virtual 167MB) edge1 Up 3 minutes 4 days ago running ```edge2 containers
```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-2 2B (virtual 40.3MB) edge2 Up 23 hours 4 days ago running diun-2 0B (virtual 54.6MB) edge2 Up 23 hours 4 days ago running duckdns 21.7kB (virtual 40.8MB) edge2 Up 22 hours 22 hours ago running netdata-2 26.6kB (virtual 394MB) edge2 Up About a minute (healthy) About a minute ago running pihole-2 128MB (virtual 422MB) host Up 23 hours (healthy) 23 hours ago running promtail-2 0B (virtual 185MB) edge2 Up 23 hours 3 days ago running scrutiny-collector-2 9.37kB (virtual 189MB) edge2 Up 23 hours 4 days ago running wireguard-2 23kB (virtual 167MB) edge2 Up 23 hours 4 days ago running ```Generally, both servers need no daily maintenance and the Diun jobs run on Sundays to let me know what containers need updating (thanks for that, btw!).
I wanted to implement fail2ban in a docker container and used this version based on the good work done with Diun. I found a tutorial and followed it, amending the configuration to suit my needs. It works, and I sometimes get ban notifications.
However, after a day or so of usage the following will occur:
- System load increases
- Disk I/O increases
- Zombie processes spike
- Threads increase
According to my netdata metrics, this is due to fail2ban. I'm just not sure how to see what it was doing at the time.
Steps to reproduce
- Start server
- Check all docker services are running
- Wait 24+ hours
- Monitor system performance
Docker info
Client: Docker Engine - Community
Version: 24.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.19.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 9
Server Version: 24.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.21-v8+
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.855GiB
Name: id-edge1
ID: 7HKR:XS7T:KP55:WXII:JDYT:SCR5:EF3N:PHZA:XOAK:EWEP:3J7W:NWFN
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker Compose config
name: fail2ban
services:
fail2ban:
cap_add:
- NET_ADMIN
- NET_RAW
container_name: fail2ban
dns:
- 192.168.1.1
environment:
F2B_DB_PURGE_AGE: 14d
F2B_LOG_LEVEL: WARNING
F2B_LOG_TARGET: STDOUT
TZ: America/Edmonton
hostname: edge1
image: ghcr.io/crazy-max/fail2ban:latest
network_mode: host
restart: unless-stopped
volumes:
- type: bind
source: /srv/fail2ban/data
target: /data
bind:
create_host_path: true
- type: bind
source: /srv/nginx-proxy-manager/data/logs
target: /var/log/npm
read_only: true
bind:
create_host_path: true
- type: bind
source: /etc/timezone
target: /etc/timezone
read_only: true
bind:
create_host_path: true
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
bind:
create_host_path: trueLogs
fail2ban | Setting timezone to America/Edmonton...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-07 15:49:10,951 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-07 15:49:10,956 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-07 15:49:10,958 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-07 15:49:10,959 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-07 15:49:10,959 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-07 15:49:10,967 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-07 15:49:10,969 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-07 15:49:10,998 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-07 15:49:11,002 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-07 15:49:11,008 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-07 15:49:11,011 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-07 15:49:11,014 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-07 15:49:11,016 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-07 15:49:11,043 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-07 15:49:11,045 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-07 15:49:11,047 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-07 15:49:11,051 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-07 15:49:11,053 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-07 15:49:11,055 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | Setting timezone to America/Edmonton...
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-08 23:00:12,148 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,158 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-08 23:00:12,161 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-08 23:00:12,162 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-08 23:00:12,162 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-08 23:00:12,197 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,201 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-08 23:00:12,254 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-08 23:00:12,269 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-08 23:00:12,283 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-08 23:00:12,285 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-08 23:00:12,289 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-08 23:00:12,295 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-08 23:00:12,356 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,358 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-08 23:00:12,365 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-08 23:00:12,379 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,381 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-08 23:00:12,397 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | 2023-07-08 23:00:18,583 fail2ban.utils [1]: ERROR 7f9a787270 -- exec: bash /data/action.d/telegram_notif.sh -a start
fail2ban | 2023-07-08 23:00:18,583 fail2ban.utils [1]: ERROR 7f9a787270 -- returned 6
fail2ban | 2023-07-08 23:00:18,584 fail2ban.actions [1]: ERROR Failed to start jail 'npm-general-f-b' action 'action-ban-docker-f-b': Error starting action Jail('npm-general-f-b')/action-ban-docker-f-b: 'Script error'
fail2ban | 2023-07-09 11:50:38,474 fail2ban.actions [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned
[repeated every 5 minutes until]
fail2ban | 2023-07-09 12:58:15,383 fail2ban.actions [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned
fail2ban | Setting timezone to America/Edmonton...
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-10 08:24:32,151 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,157 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-10 08:24:32,160 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-10 08:24:32,161 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-10 08:24:32,162 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-10 08:24:32,176 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,178 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-10 08:24:32,223 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-10 08:24:32,227 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-10 08:24:32,235 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-10 08:24:32,237 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-10 08:24:32,243 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-10 08:24:32,245 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-10 08:24:32,304 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,306 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-10 08:24:32,309 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-10 08:24:32,312 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,325 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-10 08:24:32,329 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | 2023-07-11 09:46:53,185 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.8.92 already banned
fail2ban | 2023-07-11 17:05:32,996 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned
[repeated every minute until]
fail2ban | 2023-07-11 17:10:14,200 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned
fail2ban | 2023-07-12 13:24:47,594 fail2ban.observer [1]: ERROR can't start new thread
fail2ban | Traceback (most recent call last):
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 221, in run
fail2ban | meth(*ev[1:])
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 361, in db_purge
fail2ban | self.add_named_timer('DB_PURGE', self.__db_purge_interval, 'db_purge')
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 119, in add_named_timer
fail2ban | t.start()
fail2ban | File "/usr/lib/python3.10/threading.py", line 935, in start
fail2ban | _start_new_thread(self._bootstrap, ())
fail2ban | RuntimeError: can't start new thread
fail2ban | Setting timezone to America/Edmonton...
fail2ban | ln: /etc/localtime: File exists
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-12 14:10:04,526 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,531 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-12 14:10:04,535 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-12 14:10:04,536 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-12 14:10:04,536 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-12 14:10:04,558 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,560 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-12 14:10:04,593 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-12 14:10:04,602 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-12 14:10:04,606 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-12 14:10:04,608 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-12 14:10:04,615 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-12 14:10:04,617 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-12 14:10:04,655 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,657 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-12 14:10:04,664 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-12 14:10:04,671 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,673 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-12 14:10:04,677 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
Additional info
The latest issue occurred just before 2023-07-12 01:00 MDT. The only entry in the docker logs for this day was "ERROR can't start new thread" at 13:24 MDT, likely when I ssh'd into the server to check why it was so slow.
The fail2ban container can see the NPM logs:
$ docker exec fail2ban ls -lha /var/log/npm/error.log
-rwxrwxr-x 1 1000 1000 25.6K May 26 12:11 /var/log/npm/error.logThe database appears to be valid:
$ ls -lha data/db/
total 212K
drwxrwxr-x 2 idsvc docker 4.0K Jun 27 14:28 .
drwxrwxr-x 6 idsvc docker 4.0K Jul 11 14:48 ..
-rwxrwxr-x 1 idsvc docker 204K Jul 12 14:34 fail2ban.sqlite3The issue appears to be iowait related:

Just before I rebooted the server, the number of fail2ban processes was at 1,655:

What am I missing here?