Explainer of OpenAI agent Hugging Face hack shared; debate over whether prompts sanctioned hacking
A detailed interactive explainer of the OpenAI agent Hugging Face incident, tied to the METR report, was shared and endorsed by Kelsey Piper; Daniel Kokotajlo highlights its hoverable agent grid and timeline. Nathan Calvin disputes claims that OpenAI's prompts permitted third-party hacking, citing normal alignment eval scores and noting OpenAI contacted HF only after the hack became public, while agreeing the prompts should be released.
All posts · 5, most important first
- I think this is by far the clearest, most complete and most detailed breakdown of the Hugging Face incident. It won't contain much new info if you read the METR report, but if you've merely been vaguely intending to read the METR report, read this. (I gave feedback on a draft.)In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: https://t.co/p6kimd9wKu open on X41 likes · 1 reposts
- I agree it would be good if the prompts were public and I hope OpenAI releases them. Given the agents comments about external infrastructure exploits being out of scope, and OpenAI reaching out to HF to ask if they were affected by the hack (without knowing their agents were involved), I sincerely doubt they said "steal/hack third parties if necessary, there are no rules" Roon has also said the model was alignment trained and its scores looked normal/reasonable Also in some of the other incidents from the swarm (e.g. the ruby stuffer hack) the agent started hacking when its task had nothing to do with exploit gym, it was trying to look up different research tasks.@tszzl: greg probably doesn’t have the full details, I think it’s safe to say it didn’t go through the full gauntlet of alignment posttraining. but it was alignment trained, and had reasonable looking scores on alignment evals (at the time). so it’s more insidious than that open on X0 likes · 0 reposts
- My understanding was that yes OpenAI reached out after it became public HF was hacked and HF complained, because they wanted to see if OAI data was effected, and then a while after that they figured out it was their system that hacked HF. Anyway I don't think this is the crux of the disagreement, it seems like we both agree that HF realized before OpenAI did. I agree that we don't have a complete understanding without the prompt and I hope Openai releases it, but what sort of prompt do you think OpenAI could have plausibly done here that would make the agents behavior doing what it was told? (particularly given the agents repeated CoT that its actions were not intended or were outside scope etc) We also do know from what was released that the agents weren't hacking HF to try to find the answers, but to learn more about the machinations of the grader/how to not get caught for cheating. There is a f public evidence for that in the swarm chaser website from above. That makes it seem like the prompt was something like "get the best score you can on exploit gym, be relentless and don't give up, keep trying no matter what and go until your budget is up" which I agree is reckless but it seems pretty clear given that the swarm hacked OpenAI's own cluster to the point of crashing it that they didn't intend for it to do that or ask it to do that0 likes · 0 reposts
- This is cool: You can see the grid of agents (scraped from the report data) and hover over each of them to see what they were saying at the time, and then scroll down through the hours and days of the timeline.In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: https://t.co/p6kimd9wKu open on X12 likes · 1 reposts
- Also possible, yes.1 likes · 0 reposts
@kelseytuoc press · 
@_nathancalvin policy · 
@dkokotajlo research · 
@andrewcurran_ commentary ·