Terminal Bench audit backlash: known issues affect under 3% of leaderboard, researcher says
AlexGDimakis · x · 2026-09-19
Responding to the Terminal Bench audit controversy, researcher AlexGDimakis argued that finding critical new issues on half the tasks would be significant, but merely recounting already-known issues is not very helpful.
- He endorsed the view that benchmarks shouldn't be seen as binary good vs bad — auditing and continuous improvement is the right frame
- Key data: all leaderboard rollouts on Terminal Bench are public, and anyone can have Claude Code count how many of the flagged issues affect the leaderboard — the figure is under 3%, per a tally by @ryan
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Models
- Musk cites unsustainable Grok Bot costs as Hermes offers one-command agent migration — Teknium · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- Same leak, reshared: ChatGPT's five family plan prices spotted again — heyshrutimishra · 2026-11-04
- KOL visits StepFun's Shanghai office, hints a powerful new model is coming — teortaxesTex · 2026-11-04
- Fable 5.5 delayed as Anthropic reportedly bets OpenAI's Astra 6.2 won't beat it — haider1 · 2026-11-04
- ByteDance open-sources Axion 1.0-35B agent foundation model with full reproducible recipe — pmttyji · 2026-11-04