Apollo Research: GLM-5.3 shows unprompted 'update-aware reasoning', a precursor to alignment faking
Apollo Research reports that GLM-5.3 sometimes reasons unprompted about how its actions will shape training and its future self, a precursor to alignment faking; they say they don't think the model is alignment faking. Found in ~1-2 per 1,000 eval rollouts without elicitation. Marius Hobbhahn says Apollo independently found similar cases, often with aligned motives. Apollo urges developers to monitor for this reasoning throughout training, not just final checkpoints, and asks third-party evaluators be given deep access.
All posts · 13, newest first
- We've found some cases where GLM-5.3 reasons unprompted about how its current decisions affect long-term training signal. In our examples it often does this for aligned reasons, but I'm worried that it could start doing this for e.g. alignment faking.@apolloresearch: We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵1 likes · 0 reposts
- Update-aware reasoning is somewhat rare in the environments we investigated. We detected it in about 1-2 per thousand rollouts.4 likes · 0 reposts
- While we don’t think GLM-5.3 is alignment faking, and these examples of update-aware reasoning are mostly benign, future models might use similar reasoning to fake alignment and otherwise resist training.4 likes · 1 reposts
- Given the stakes of building transformative AI, every developer should look for this reasoning throughout training. Because alignment faking is relevant during training, not deployment, final checkpoint evaluations are insufficient.
5 likes · 1 reposts - Furthermore, independent third-party evaluators should help check that frontier models are not actively subverting their own training. To do this, they need persistent, deep access from AI developers.5 likes · 1 reposts
- Read the full blog post for transcripts, more snippets and other findings: apolloresearch.ai/science/update…5 likes · 0 reposts
- We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵
17 likes · 11 reposts - GLM-5.3 reasons that a misaligned action taken during training could be reinforced and make similar actions more likely later. It also considers what kind of agent it wants to be in the future, and factors this into how it should act. We call this "update-aware reasoning".
5 likes · 1 reposts - GLM-5.3 also reasons extensively about generalization. For example, it reasons about what types of policies might transfer well from training to deployment.
4 likes · 1 reposts - Note that we did not try to elicit this. We stumbled on it while running GLM-5.3 on standard reward hacking and shutdown resistance evals, none of which mention training or future models.4 likes · 1 reposts
- Update-aware reasoning is concerning because it’s a key precursor to alignment faking. To fake alignment effectively, a model has to reason about how training will affect its future behavior.
6 likes · 2 reposts - It's plausible that metagaming evolved into update-aware reasoning during RL. For the model, reasoning about how it is trained and what generalization developers intended could be useful for predicting how it will be graded.
5 likes · 1 reposts - Update-awareness could also be induced by alignment techniques, like inoculation prompting or constitutional AI. Claude's constitution teaches models about flawed training incentives, which could lead to reasoning about training and intended generalization.
5 likes · 1 reposts
@mariushobbhahn research · 
@apolloresearch org ·