Terminal-Bench 4.0 scoring flaws hit <3% of rollouts, despite Epoch finding 30/66 defective tasks
DimitrisPapail · x · 2026-09-19
Epoch's review found 30 of 66 Terminal-Bench 4.0 tasks have scoring defects (exploitable graders, answer leakage, correct solutions rejected), sparking "TB 4.0 is broken" takes.
@DimitrisPapail argues Epoch's framing, while not literally saying broken, implies distrust. @ajratner adds key context: the terminal-bench team flagged these issues themselves as PRs (how Epoch found them), and empirically they affect <3% of leaderboard rollouts — well within reported CIs.
Consensus: exploitable graders need fixing, but "30/66 tasks have issues" ≠ "45% of results are wrong" — exploitability doesn't tell you how often it happened. The team is advancing continuously improved open benchmarks rather than shipping a broken one.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04