Audit Sparks Debate Over Terminal Bench 4.0 Flaws
An audit flagged 45.5% of Terminal Bench 4.0 tasks as broken, and Epoch found scoring flaws in 30 of 66 tasks, but maintainer AlexGDimakis countered that known issues affect less than 3% of the leaderboard, fueling debate over the benchmark's integrity.
2026-09-19 ~ 2026-09-19 · 3 related posts
- Episode 1: Epoch AI's First Benchmark Audit: Only 4 of 15 Pass, 9 Found Flawed(2026-09-18, 16 posts)
- Episode 2: Audit Sparks Debate Over Terminal Bench 4.0 Flaws(2026-09-19, 3 posts)
- Audit flags 45.5% of Terminal Bench 4.0 tasks as broken; maintainers push back on rigor — DimitrisPapail · 2026-09-19
- Terminal-Bench 4.0 scoring flaws hit <3% of rollouts, despite Epoch finding 30/66 defective tasks — DimitrisPapail · 2026-09-19
- Terminal Bench audit backlash: known issues affect under 3% of leaderboard, researcher says — AlexGDimakis · 2026-09-19