"oh they should have known to run it sandboxed". Sounds like pretty shit, untrustworthy software when you put it like that. I mean, we've been running java applets downloaded from the Internet in sandboxes for what, 30 years? But that's because it's an obvious attack vector. You're saying this trillion dollar industry of the future produces stuff of a similar quality as we sandboxed against in the 1990s.
And that you aren't employing people who are skilled enough to build sandboxes for your own products. Great industry.
Sure sounds like a company who don't think they need to do things like "obey laws" or "exhibit at least the sort of care you would expect from a second year undergrad". Tremendous industry, apparently non-consensual nudifying is a free speech issue too?
https://www.wired.com/story/openais-rogue-ai-agent-hacked-more-than-just-hugging-face/
oh no what is to be done, if only there were laws against things like 'breaking into someone else's computer without permission'.
oh huh there are. in fact quite a lot of them. oddly enough "we knew it might be successful in breaking into someone else's stuff, but made sure to not watch when it was running" isn't a defense.
Are laws like the CFAA used in bullshit ways? Absolutely, RIP Aaron. Is this one of those cases? Absolutely not.
So Karl Bode's scathing write-up of the "OpenAI rogue hacker robot" marketing stunt highlighted the under reported detail that this extremely responsible AI company ran their exploit kit with safety guardrails disabled. Once again, I ask: why haven't they been charged? Do we want to see more of this "it's not a crime if the software is AI shaped"? I think we should try and discourage it.
They setup a bot specifically to do red team stuff and turned it loose on their competition. Knowing they can just do what Anthropic does with every new model and claim it’s “dangerous” for that free marketing bump in the press.
@swearyanthony aaaargh, no, Bode's take is bad. They disabled the guardrails so they could test the model on the ExploitGym benchmark, but instead of hacking the target system it executed a much more complex hack against Hugging Face. Which, as we've discussed before, would be criminal if a human did it and should be criminal in this case too. But more importantly, the AI did something that OpenAI very much did not want in order to satisfy a narrow definition of success: a classic alignment failure.
@swearyanthony I agree that they should not have run this experiment, though! It's wildly irresponsible if their alignment techniques are this fragile, and they should have been able to find that out in a lower-stakes way.
@pozorvlak that they had disabled some guardrails definitely wasn't something their initial "oh no the AI robot is too powerful for us little humans" said
@swearyanthony it's mentioned in the first paragraph of their blog post: https://openai.com/index/hugging-face-model-evaluation-security-incident/ (skip past the updates from July 28 and 29)
@swearyanthony so you either have to intend the crime or be reckless as to a foreseeable risk and I think their defence here is going to be 'this is a new technology and we didn't frankly know it could DO that'
@onekind they *designed* a program to break into other computers, then ran it in a way where it would do that, and claimed they didn't bother to monitor what it was doing.
@swearyanthony You would still need to use POSIWID intentionality and I'm not sure where American jurisprudence is sitting with that.
Jul 29, 2026, 2:18 PM@swearyanthony because they're so advanced they're unstoppable and we just serve the models now
@swearyanthony they did run it sandboxed. It found a 0day and broke out of the sandbox.
(This is not an argument against them facing criminal accountability: quite the reverse)
@pozorvlak yes as I said elsewhere in the thread - them not being willing to ensure it was properly sandboxed is a "them" problem.
@pozorvlak why wasn't there monitoring? If it was for some reason impossible to do even *that*, why wasn't the project stopped for a rethink.
@swearyanthony It wasn't stopped because they didn't know the attack was coming from them until after it was over, and they didn't expect anything like this to happen. Did they need better sandboxing and monitoring? Sure. But it's going to be increasingly difficult to defend against AIs as they become more capable, and ultimately I don't think they shouldn't be doing this kind of gain-of-function research at all. Criminal penalties might help focus their minds.
@swearyanthony apparently they *did* stop training after the hack was discovered: https://www.astralcodexten.com/p/highlights-from-the-discourse-on (under the subheading "Sam Altman"). And according to the July 28 update to https://openai.com/index/hugging-face-model-evaluation-security-incident/, the model involved has been "deactivated, encrypted, and restricted... from research access."
Two cheers for OpenAI, I guess?
@pozorvlak @swearyanthony The AI breaking out of their sandbox should be a 5 alarm fire that you shut everything down to fix, then again a "10% chance of human extinction" is apparently an acceptable risk and that anyone can say that with a straight face makes me feel like either the world or I have gone mad
@fl0und3r I feel exactly the same! @swearyanthony
@fl0und3r @pozorvlak @swearyanthony Now ask how you can reliably detect it (reference to Rice's theorem optional)...
@swearyanthony Same for "prompt injections". Copilot might, in some circumstances, interpret parts of the Word doc it works on as its user's instructions? Strong https://en.wikipedia.org/wiki/Melissa_(computer_virus) vibes…