Fangcun AI Safety Evaluation of DeepSeek V4.1 Spurs Debate on CoT Robustness
Daniel Kokotajlo questions the robustness of Chain-of-Thought (CoT) monitorability findings from the Fangcun AI evaluation of DeepSeek V4.1. While the evaluation noted models are less capable without reasoning and typically refuse CoT manipulation, Daniel Kokotajlo requests evidence of what would happen if the model attempted to subvert its CoT.
All posts · 1, most important first
- Thanks for doing this! I'm very happy to see the discussion of situational awareness and chain of thought monitorability. Please forgive my ignorance for I have only skimmed the paper so far, but it seems that you evaluated CoT monitorability by (a) prompting the model to manipulate its CoT, and (b) turning off reasoning and seeing how capable it was without reasoning. And you found that it's much less capable with reasoning off and that it typically refuses to manipulate its CoT. That's nice, but... what if it didn't refuse? I'd like to see evidence of the form "Even if it tried its best to manipulate the CoT rather than refusing, it still can't do it." Such evidence could perhaps be provided by fine-tuning the model to manipulate its CoT, to overcome the refusal instinct. Also, relatedly, do you think the model was intentionally trained to refuse to manipulate its CoT, or is this behavior emergent?5 likes · 0 reposts
@dkokotajlo research ·