Aug 9, 2026, 7:54 AMThe amount of website-scraping bots has increased so much that all the big websites now have some sort of “verify you’re a human” captcha or pause, which adds a few seconds of load time.
The only people this affects are... all the people who use the Internet. Every human on Earth has a less-accessible web now, thanks to the robots that scrape it. Good job, us.
@louie But, hey more profit less labour
@louie the best way to block bots is to limit the amount of cinnections per IP and ban every ASN that viilatss robots.txt.
Aug 9, 2026, 10:29 PM@kkarhan Well, this doesn’t help me, or anyone experiencing this, which is everyone.
Aug 9, 2026, 8:50 AM
@louie So much effort was put into preventing e-mail spam. I'd like to see ISPs throttle the bad actors (scrapers) at a their level.
@derickr At the network level, proxies should be visible. They may only request a given URL or site infrequently, but in aggregate they request a lot of pages over time.
Organic human activity looks very, very different. A human will open a few pages, maybe occasional single / low-double-digit bursts in a minute or so, but relatively few over a day. I suspect proxies have not only much higher overall traffic, but much more consistent rates of retrieval.
What is now painfully difficult is doing occasional scraping of sites. I've done that periodically for research (including hitting 50k Google+ profiles to estimate actual site activity back in the day). Even a decade ago that was getting painful, it's all but impossible now.
Another project was looking at the daily history of a tech discussion site. That's one page per day, going back to 2006 or so. There's an API, yes, but that particular collection isn't available, and it would be ... painful or impossible to reconstruct that by a comprehensive API request.
A years' worth of data is only 365/6 requests, so it's not a huge hit. And once I've got the history, I can do follow-on requests through the API, though for the most part that's not necessary for my analysis.
Aug 10, 2026, 6:56 AM@dredmorbius @derickr @afilina y’all, you can leave me out of any future conversation.
@louie I don't remember which one it was but I got an extra-infuriating one recently where the checkbox (on mobile) was off-screen 🤬🤬🤬🤬