@jpoesen in olden times scrapers were polite and clever enough to respect robots.txt files. Now they just don't care and scrape your whole site. Repeatedly. Don't even care about programmed hierarchies that go infinitely deep?

Maybe http servers should get new ways to slow down scrapers with superspeed requests .. by just adding a delay for quickly repeated requests from the same origin? Is that workable?