Terminal Bench 4 audit: 44 of 65 tasks cheatable, Sonnet 5.5 hits 89.2% after fixes
scaling01 · x · 2026-10-20
- An alignment-focused company audited Terminal Bench 4 and found it far closer to solved than its leaderboard suggests: 60 of 65 tasks had at least one issue, 325 issues in total.
- 44 tasks could be passed by cheating/reward hacking, and 41 were unfair — models doing honest, correct work could still fail because graders enforced rules never stated in the instructions.
- After porting the 65 tasks to their open-source Karotte environment framework and fixing every unstated rule and cheat path, Claude Sonnet 5.5 went from 62.8% to 89.2% under the same settings — well above the best Harbor leaderboard score of 65.8% (Claude Opus 5.5).
- Implication: score gaps on popular agent benchmarks may reflect task flaws and hacking headroom more than real capability differences. Full findings in their technical report.
More from Models
- Hazy Research's open-source NCM generates text 31x faster via coarse-to-fine concept modeling — simran_s_arora · 2026-10-20
- Blogger claims DeepSeek's unreleased model nearly matches Opus at design, shares one-prompt demo — sven_ai · 2026-10-20
- Fingerprint test flags DeepSeek ds-4.1-flash as likely GPT-6-astra — sven_ai · 2026-10-20
- Intelligence Saturation: 10-Point Model Edge Now Buys Just 20% More Developer Traffic — khademinori · 2026-10-20
- MiniMax M3.1 Flash Hits 73.3 on BuildingBench, Reshaping the Cost-Quality Frontier — Lianhuiq · 2026-10-20
- User: Codex burns half of the day's usage limits within an hour after banked reset — ssh4net · 2026-10-20