Recent hacks spark Ball–Kokotajlo–Krueger debate over diffusion, restriction, and stopping AI training
David Krueger accused Dean Ball of misleadingly neglecting the option to stop building AI, adding a third option to Ball's diffusion-vs-restriction framing of this week's AI-driven hacks. Ball replied that little of what's happening reflects misalignment—mostly white hats discovering vulnerabilities or people hacking with models. Ball, Nathan Lambert, and Samuel Hammond also debated whether labs' earlier risk warnings were vindicated or mistimed.
All posts · 31, most important first
- Shame on Dean for continuing to misleadingly neglect the obvious option to stop building AI. I'm not surprised given his history of visciously attacking the character of anyone who suggests we have or use this option.@dkokotajlo: Yes. note that there's also a third option, which is "if it's too dangerous for everyone to have it, it's too dangerous to train in the first place." open on X5 likes · 0 reposts
- Alignment is extremely important, but very little of what we are seeing this week is because of misalignment. It’s either white hats using models to discover/appropriately disclose vulnerabilities or people hacking with models.5 likes · 0 reposts
- Everyone should be looking for data points to advance the longstanding debate in AI policy between “diffuse dual-use capabilities rapidly so the good guys can defend themselves” and “restrict capabilities proliferation to only carefully vetted good guys.” We are putting the former position to the test as we speak. No motivated reasoning to reach your preferred conclusion; just pay attention and update as appropriate in whatever direction seems true. This is much too important for low-fidelity tribal polarization.27 likes · 6 reposts
- Yes. note that there's also a third option, which is "if it's too dangerous for everyone to have it, it's too dangerous to train in the first place."@deanwball: Everyone should be looking for data points to advance the longstanding debate in AI policy between “diffuse dual-use capabilities rapidly so the good guys can defend themselves” and “restrict capabilities proliferation to only carefully vetted good guys.” We are putting the former position to the test as we speak. No motivated reasoning to reach your preferred conclusion; just pay attention and update as appropriate in whatever direction seems true. This is much too important for low-fidelity tribal polarization. open on X2 likes · 0 reposts
- I think it’s pretty clear that the hacks we are seeing are largely being done with Chinese open-weight models. The timing is too perfect. If you believe the former view—diffuse broadly so the good guys have unrestricted means of self-defense, clem delengue thought—this was part of your expectation all along. If that view is right things will shake out in favor of defense pretty quickly. It could very well be correct; I reserve judgment but certainly hope the former view is right.1 likes · 1 reposts
- To be frank: I think the magnitude was too loud given the evidence especially in 2023-2024, the labs cultivated a somewhat narrow minded safety posture, and many other small-medium decisions that got us here today. It's not going to come down to one senate hearing here or there, but a pattern that mistimed concern with risk.3 likes · 0 reposts
- We can agree to disagree, but generalizing from being right about cyber to being right about everything else isn’t supported by the evidence. Working on a project specially comparing projections of what will happen in X time by lab leadership vs what happened and the track record isn’t great3 likes · 0 reposts
- AI safety people have been warning about this for a decade! The “we” is much bigger than the labs. The criticism of the labs is fair enough, but that doesn’t mean the people who should have known better and nonetheless dismissed these concerns for years are above reproach.1 likes · 0 reposts
- This encompassed both those who said ‘we are building the cybersecurity vulnerability exploiter from the cautionary tale…’ and then those who were saying ‘here is a cautionary tale saying we should not build the cybersecurity vulnerability exploiter.' Same reactions.@deanwball: We tried to warn the world and far too often, to be frank, we were yelled at for doing so. “Regulatory capture,” “marketing hype,” “doomer anxiety,” “fear-mongering about open source,” “why don’t you talk about the benefits?” open on X2 likes · 1 reposts
- what a shocking level of delusion you must live under to believe that the cyber events of this week are not ai driven. in a few years you’ll be saying, “these vulnerabilities in the human body have always been there, ai is just exposing the underlying weakness of flesh.”0 likes · 0 reposts
- this line of criticism frustrates me a bit. we have repeatedly said catastrophic risks are on the horizon “over the coming years”, and we have been completely correct. should we not discuss a catastrophic risk unless it’s <1 year away?6 likes · 0 reposts
- It wasn't organic. Big money went to some of the most unscrupulous, low IQ people in tech advocacy to deflect concerns over cyber and CBRN risk in favor of empty abstractions (ludditism vs "permissionless innovation") and other red-herrings ("but AI didn't replace radiologists!")@deanwball: We tried to warn the world and far too often, to be frank, we were yelled at for doing so. “Regulatory capture,” “marketing hype,” “doomer anxiety,” “fear-mongering about open source,” “why don’t you talk about the benefits?” open on X5 likes · 0 reposts
- This seems right to me, unfortunately, and it is essentially a species of interest-rate rise (the opportunity cost of capital is increasing). It is true that forward and futures markets for compute, and making compute into a financial commodity, would help here.evanjconrad.com/posts/end-of-s… open on X4 likes · 1 reposts
- We tried to warn the world and far too often, to be frank, we were yelled at for doing so. “Regulatory capture,” “marketing hype,” “doomer anxiety,” “fear-mongering about open source,” “why don’t you talk about the benefits?”yes. it was always going to be this way. we tried to warn the world. open on X4 likes · 2 reposts
- But like, frontier labs did in fact have a unique vantage point and access to information, which have so far turned out to be basically right! I think things just tend to polarize now; it’s not clear to me labs in particular have nudged it along3 likes · 0 reposts
- I don't disagree with the warning, I disagree with how it was done, which came across a lot as "trust us we are the ones who know" and other factors which nudge in polarization of the discourse3 likes · 0 reposts
- so, 36 months was too soon to warn about catastrophic risk potential that we easily foresaw, and we should have just not said anything for two years and then told everyone sooner to when we were sure the risks would materialize? What about the world would be better if we had done that? I don’t get it. Is there a single example of a prominent lab employee saying catastrophic risk would manifest itself before roughly now? I can’t remember anyone serious at a lab, in 2023, being like, “it’s all gonna happen next month!” I recall 2-3 years being the common line from lab execs in 2023, which seems… exactly right? Inconvenient truth perhaps, but still truth.3 likes · 1 reposts
- setting aside the headaches, though, what should we have done? should dario and sam have refused to go to the senate in 2023 and say, broadly, “yeah, our sense is bad stuff on misuse and alignment could begin in a couple years or so”? I don’t understand what people want from us3 likes · 0 reposts
- yeah i do, is a very messy human-nature-causing-problems parable, which is like all of history2 likes · 0 reposts
- (This is one of the many ideas in the ai action plan, and broadly speaking the plan is full of ideas intended for the mid-game of this whole transformation, just like this one. Imo, in terms of substance aiap, as a 15 month old doc, reflects a well-grounded model of the future)2 likes · 0 reposts
- AI is clearly not a "normal technology" or merely a tool. These are digital minds that functionally emulate brains. As a comp functionalist, I think that's just true and philosophically interesting, but mostly orthogonal to cyper / CBRN misuse risks.1 likes · 1 reposts
- What are you refering to? I'm also not putting people into camps; the actors I'm refering to were largely pay for play influence ops. My longstanding message has been that doom / bloom, go slow / go fast groupings are unhelpful false dichotomies. If anything it's these influence ops that have tried to force tribal issue polarization.1 likes · 0 reposts
- We will see what happens, but a great many people mocked these risks, attributed them to ludditism, or otherwise distracted the public from the truth. This was bad.1 likes · 0 reposts
- this is a meme that is importantly untrue; openai did in fact release gpt 2. but the concern was that the marginal risk of broad diffusion goes up with time. we are undoubtedly seeing the effects of that higher risk in the world now, but we don’t know how it will shake out. there are people at openai who predicted this dynamic 7 years ago and wanted the organization to build muscle memory about how to weigh the tradeoffs and make those judgment calls. the people you’re dismissing as worry warts foresaw what would end up being one of the most sensitive and consequential decisions that a frontier lab makes.1 likes · 0 reposts
- but just like the bubble we're in, timing matters a ton when you're asking for societal response. if you get timing wrong, you literally get a repeat of the boy who cried wolf story!1 likes · 0 reposts
- cyber was right. bio will be correct in terms of capabilities. job stuff, too early to say but would be bad for people in positions of power to encourage complacency, so it’s worth talking about despite the deeper levels of uncertainty.1 likes · 0 reposts
- “we” here includes the broader ai policy world too. most of the time when I personally warning people, I was a policy researcher at a think tank or in government, not a lab.1 likes · 0 reposts
- The reason people bring it up is because it’s caused a lot of headaches to other communities on the policy front. I agree it’s a mess of dynamic, just trying to explain. I don’t think bio risks and job loss risks have panned out yet, so it’s tricky.1 likes · 0 reposts
- We tried to warn the world and far more often, to be frank, we were yelled at for doing so. “Regulatory capture,” “marketing hype,” “doomer anxiety,” “fear-mongering about open source,” “why don’t you talk about the benefits?”yes. it was always going to be this way. we tried to warn the world. open on X1 likes · 0 reposts
- oh yes1 likes · 0 reposts
- Part of the problems is these companies have forecast major problems many time and this is maybe the first time it showed up0 likes · 0 reposts
@davidskrueger research · 
@deanwball policy · 
@dkokotajlo research · 
@natolambert research · 
@thezvi commentary · 
@hamandcheese policy ·