Since yesterday, I've blocked nearly 100 IP addresses which had requested a hidden directory blocked by robots.txt - the graph already shows a dramatic fall in data transfer.

B⁶ is, as the name suggests, a blunt tool. But it's effective. I'll release the source code once I've tidied it up.

Some bots, like intricle-image-extractor are obvious scrapers, but most spoof common browsers. I half wish OSes would cryptographically sign their requests so you knew it came from a genuine device.