Critics flag flaws in TerminalBench 4.0; fixed version shows Sonnet 5.5 near saturation

3 people · 3 posts · first post Mon 16:52 UTC · latest 5 days ago

Critics raised concerns about flaws in TerminalBench 4.0, and a subsequently fixed version reportedly shows Sonnet 5.5 near saturation on the benchmark. Benchmark integrity issues like this bear directly on how capability and safety evaluations are interpreted; further details from the original posters were not available in the posts shown.

All posts · 1, most important first

1. @tim_hua_ research · Mon 18:48 UTC
   Sounds about right
   We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts. open on X
   0 likes · 0 reposts

Reposted by @dkokotajlo, @kelseytuoc.