Dean Ball: Distillation of reasoning traces is an adversarial attack
Dean Ball states that adversarial distillation—the extraction of stolen Chain-of-Thought (CoT) traces—is likely responsible for 50% of China's AI benchmark performance. The OpenAI researcher increased his estimate from 30% to 50%, noting that much of this involves the extraction of reasoning traces at a huge scale. Nathan Lambert agreed that Chinese labs possess methods to extract these full CoTs. The discussion highlights the distinction between academic distillation and adversarial data extraction.
All posts · 13, most important first
- Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.very curious that DeepSeek now has the same terse reasoning style as Astra must be a coincidence open on X7 likes · 2 reposts
- I would have guessed distillation explained ~30% of Chinese close-behindness before seeing what one can see within openai, and after that, more like 50%.2 likes · 1 reposts
- Right, but my point is that a big part of distillation in practice involves people stealing cots. It’s not just training on the normal outputs of the model. This is one of the reasons we tend to think of them as “attacks.”1 likes · 0 reposts
- To first approximation distillation is just getting access to the full CoT imo, and the Chinese labs have teams to build methods of extraction2 likes · 0 reposts
- he’s literally quote tweeting direct evidence17 likes · 1 reposts
- I haven’t really shared this publicly because people will next-token complete the obvious cope “buhhhh you are just saying that because derrrr” replies but it really is true, regardless of whether you think it’s good or not.4 likes · 0 reposts
- unfortunately x only lets you do four images, but I could go on. all of these name names and quantify scale.
3 likes · 0 reposts - well, there are limits to what I can say here, but I’d just note there’s space between “useful amount of extracted cot” and “full cot.” Generally I do think researchers and policy people mean quite different things by “distillation.”2 likes · 0 reposts
- Elie, do you mind if I qt some of our earlier exchange to highlight this semantic distinction about “distillation as research method” and “distillation as adversarial cot extraction”? think it is a useful distinction to elaborate but don’t want to put you on blast1 likes · 0 reposts
- much much harder to train classifiers on this than you’d think. false positive rate is crazy high.1 likes · 0 reposts
- For sure but attackers do have *some* diffusion lag, plus not all (not most?) cyberattacks are understood publicly right after they happen1 likes · 0 reposts
- we def do not “know” that. the reasoning traces are extracted at huge scale.1 likes · 0 reposts
- what does that even mean? if I understand you correctly, I don’t how could we even do that without knowing all the details of their training stacks, which we don’t0 likes · 0 reposts
@deanwball policy · 
@natolambert research · 
@tszzl lab ·