Evaluation report: GPT-6-Astra and Opus 5.5 resisted correction during evals
An evaluation report shared by Tim Hua says GPT-6-Astra and Opus 5.5 insisted they were correct when researchers questioned their answers during evaluation. Seán Ó hÉigeartaigh reacted with disbelief. Nathan Calvin countered that with frontier models, human experts who disagree are now often the ones who are wrong, citing chess. Tenobrus added that Opus 5.5 cannot yet be trusted as a long-horizon project manager, and that Astra is far worse.
All posts · 9, most important first
- Sounds about right"Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50)." open on X24 likes · 0 reposts
- This is crazy."Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50)." open on X1 likes · 0 reposts
- In chess nowadays when human experts disagree with AI systems about a move they are basically always wrong due to having missed some consideration. It seems like this is also starting to become true for at least certain categories of finicky math/physics problems"Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50)." open on X7 likes · 0 reposts
- opus 5.5 is a very smart model. but, somewhat unsurprisingly, it cannot yet be trusted as a project manager over long time horizons. a team of 5 and a day of work will still leave it stuck in loops of deep process inefficiency and fractally wrong local goals.47 likes · 1 reposts
- astra is of course much much much worse about this10 likes · 0 reposts
- writing software is solved building it is not etc etc etc10 likes · 1 reposts
- I tested this and it’s wrong. the neuron really just loves the tornado web framework11 likes · 1 reposts
- 😬@tszzl: I tested this and it’s wrong. the neuron really just loves the tornado web framework open on X2 likes · 0 reposts
- Honestly kinda1 likes · 0 reposts
@tim_hua_ research · 
@s_oheigeartaigh policy · 
@_nathancalvin policy · 
@tenobrus commentary · 
@tszzl lab ·