Report: TerminalBench 4.0 has issues; Sonnet 5.5 scores 89.2% after fixes, mostly saturated

3 people · 3 posts · first post Mon 16:52 UTC · latest 3h ago

A report says TerminalBench 4.0 had issues; after fixes, Sonnet 5.5 scores 89.2%, suggesting the benchmark is largely saturated. The posts available are reposts with no substantive commentary, so the specifics and safety relevance remain unclear.

All posts · 1, most important first

1. @tim_hua_ research · Mon 18:48 UTC
   Sounds about right
   We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts. open on X
   0 likes · 0 reposts

Reposted by @dkokotajlo, @kelseytuoc.