I have a VERY stupid idea to stop LLM scrapers: You send the website as a torrent. You only send it to people willing to seed it.
Then the LLM bots can send the page to one another, and there are so many of them that they'll also help reduce hosting costs.
(Inspired by @linear 's issues accessing documentation)
Wait.... Is this a bad idea? Isn't good basically just a CDN but not owned by a giant company?
Obviously it would work a lot better for normal HTML/CSS then database stuff, but high l honestly that seems like a good thing to me...
@Canageek @brooke remember to consider the possibility of updates to the document(s) as you ponder this, since old torrents will circulate ~indefinitely, but web pages often change frequently.
The #SciOp project has been working at the intersection of those two ideas, but I would be lying if I said I fully understand how mutable torrents work, or what they've layered on top of that to make it practical for their goals.