And the comminity blocklists are updated when more than a couple ( I think the number is something like 10-50 ) instances of crowdsec block an ip in some fast timeframe.
The ai blocklist just adds IP when even one instance finds an AI trying to scrape right from the useragent.
So even if the community blocklist has fewer ai ip’s, it does eventually include them.
Try crowdsec.
You can set it up with list’s that are updated frequetly and have it look at caddy proxy logs and then it can easilly block ai/bot like traffic.
I have it blocking over 100k ip’s at this moment.
I use nixos on my desktop, the server is a debian one but might be good to install nix on it.
I just have to import only one. Might just use thunderbird for that.
Will test out mailcow and see how it goes.
I got tagged in one called “N***er balls”.
Imagine waking up and seeting that email notification.
PS: Picture just to confirm:
NOTE: I have also seen since yesterday that my crowdsec instance has been blocking way more ip’s for bots trying to crawl and shit like that so I think that this all might be a more general uptake in bots and that sites that never really had great protections against it are now taking the fall.
Just from yesterday to today I got over 100k more blocks.
I’m using the default list alongside Firehol BotScout list and Firehol cybercrime tracker list set to ban.
Also using the Firehol cruzit.com list set to do captcha, just in case it’s not actually a bot.
I’m also using the cs-firewall-bouncer and a custom bouncer that’s shown on crowdsecs tutorials to detect privilege escalation for if anybody actually manages to get inside.
Alongside that I’m using a lot of scenario collection’s for specific software I’m using like nextcloud, grafana, ssh, … which helps a lot with attacks directly done on a service and not just general scraping or both path traversing.
All free and have been using it for a year, only complaint I have is that I had to make a cronjob to restart the crowdsec service every day because it would stop working after a couple days because of the amount of requests it has to process.