one thing about making a Django site is that dealing with these residential proxy scrapers is tough. It’s definitely easier to handle with a Go server that’s faster or a static site. The site doesn’t even have anything useful to scrape!
I don’t really want to use Anubis since it’s a bit annoying for users so I’ve been doing a lot of aggressive IP blocking but it doesn’t feel great.
(not really looking for advice atm)
@b0rk If you find a good solution, I'd love to hear it!
I haven't been checking logs or anything, do the residential proxy scrapers individually hit the site a lot of times in a row, or just a few times from each one, times a lot of separate IPs?
@b0rk What are the problems you've found for users imposed by Anubis? I've never used it.
Oct 1, 2026, 2:38 PM@b0rk This speaks to my current feelings too. It’s rough.
@b0rk i moved to static site with Nepenthes for catching in endless maze the crawlers/scrapers not respecting robots.txt. It's been working well for several months. But I understand static page isn't always feasible
@lpryszcz yeah most of my sites are static and it’s great