Oct 16, 2026, 11:03 AMToday's badly-written residential proxy AI crawler fun: 5.8 million requests from 2.3 million unique IP addresses in 24 hours for a site with 12,000 static pages that hasn't changed since 2003. Most of these requests are from consumer networks and pretending to be real browsers. As many other folks have noted, it's becoming increasingly difficult to mitigate the impact of these attacks.
@beasts any advice on mitigating these? Our small family business is an e-commerce store with over 10,000 of our badge designs added over 20 years. Self hosted. We’ve tried cloudflare but never quite sits right and always hurts our rankings. We get bombarded daily. Imunify360 has helped but it’s like the hoardes.
Oct 16, 2026, 12:10 PM@jodsclass no easy solutions, I'm afraid. The plague of residential proxies makes IP-based filtering ineffective, and the bots are increasingly good at impersonating legitimate users. Best thing you can do is serve as much as possible statically, although for the volume we've seen over the last 24 hours, even that isn't enough.
Oct 16, 2026, 12:20 PM
@nono @jodsclass @beasts I've found that a lot of bots hit lots of invalid pages as well. If you have enough control and can use Fail2ban then you can block a lot of them after they first request a dot-file (like .env) or a bunch of config file paths.
Doesn't help for lots of the initial bots, but it's amazing how many proxy IPs it does catch.
I've also put something like the session cookie thing on my dynamic site. No referer? You get a simple "sorry, bots are ruining everything" message and a link back to the page that should then give a referer.
Doesn't even need to be a legit referer. I don't check that it's valid. Just that there is a value. Because the number of people hitting my site not via a link is tiny. And for now many of the bots don't pass a referer. Legit bots get a pass based on user agent string matching.
Oct 16, 2026, 2:57 PM@ibboard @nono @jodsclass Unfortunately 5.8m requests from 2.3m IPs means that even if we can identify them on the first request, blocking the IP doesn't achieve much.
Oct 16, 2026, 4:33 PM
@kasperd @jodsclass @nono @beasts I've considered that for my servers. I block all of China because it's basically 99.9999% junk (and I'm not convinced that most of the real users will get out past the Great Firewall). And I've started blocking ranges for AWS, Google Cloud, etc
But it feels bad that we're at the point of having to block every residential connection in Vietnam, West Africa, and sometimes even the USA just because of this abusive behaviour. Residential proxies have huge scope for collateral damage (which is why they do it).
If governments don't start legislating against it then we'd need big websites to start putting in "you're blocked because your IP has been seen hosting a botnet - here's how to check whether your TV is doing stuff you didn't know about" messages on their sites to start to turn things around. Or maybe CloudFlare needs to start doing it for their customers (even for legit connections, if the IP has a bad rep)
Oct 16, 2026, 5:51 PM@zip @beasts Subnets are a nightmare and need a separate database. When I'm manually adding abusive blocks then I do a lookup on the IP and check its subnet, then check adjacent blocks in case they're the same provider. I've got everything from /10s to /24s in the list. Mostly /19 to /24s.
And sometimes a company has consecutive /17 through to a /22 ranges because the final /22 of the /16 range was sold off to someone else at some point (or maybe they bought it up piecemeal - because it is normally a cloud provider or other large network who might buy up IPs as IPv4 ran out).
I wish there was an AS-based way to do it, but I haven't found a way to do it yet. Even the WhoIs sites that list ASNs don't always list an IP range/ranges for the block.
Oct 16, 2026, 6:52 PM@ibboard @beasts aye. I've poked at this too and it looks like a real mess. And it still assumes that the traffic is nonuniform relative to the baseline, and I've a horrible feeling that it isn't.
I guess the next step would be to build that hilbert curve visualisation of IP space to see what jumps out.
… I really should have gotten a job at CloudFlare or Fastly, I'd probably absolutely love and hate trying to solve this for a living
@beasts sounds like a great backup strategy
Oct 16, 2026, 4:33 PM@rakoo we've not, but it relies on being able to successfully identify the bots. Looks like it uses a list of AI bot user agents -- that definitely doesn't work.
The traffic volume we've seen is now a problem for purely static sites. We haven't got the resources to feed them generated garbage, even if we could identify them.