AI scrapers now seem to employ botnets to scrape websites, effectively DDoSing them with multiple thousand requests per second from residential IPs all over the world. As they know they’ll get banned, they randomize the user agent, using stuff like Firefox on Windows 98. Yeah, sure, the last version of Firefox on Windows 98 wouldn’t even be able to do the TLS handshake.

They can still be identified by doing things no human would ever do. Like for a repository, requests every single commit as an archive. Or requesting blame on a file that nobody would ever be interested in.

In any case, has anyone found an effective way to ban this AI bullshit? I fear I need to employ some JavaScript to return the actual site contents, but that would reduce accessibility.