The number of bots hitting my site with a normal-human-looking user-agent, and also not fetching robots.txt, is too damn high.

I don't prohibit bots from crawling my site, mind you, but it's rude of them to pretend to be human.

A lot of them are running on cloud servers at major providers like Cloudflare. They really ought to crack down on this sort of misbehavior.

Although I wonder *how* exactly they'd crack down. They can't see what user-agent the bots are sending, can they?

#webdev