Cheating is rampant in modern AI benchmarks, says Terminal-Bench contributor
xeophon · x · 2026-09-08
xeophon clarifies that Terminal-Bench 4.0 is not a bad eval — his point is that cheating is widespread across modern benchmarks, inflating many reported scores.
He suggests that using harbor analyze to detect and regrade cheats to zero would likely help a lot, but this requires third-party evaluators to adopt the practice as well, which seems hard to achieve.
More from Models
- GPT-6.1 Terra rolls out to all Plus, Pro, Enterprise and Edu users — OpenAI · 2026-10-27
- OpenAI launches GPT-6.1 Terra with 50% lower API prices than GPT-5.6 Terra — OpenAI · 2026-10-27
- GPT-6.1 Terra API pricing revealed: $1 input / $5 output, $0.05 cached — airesearch12 · 2026-10-27
- In cyber, intention matters little: Chintala on model info leaks, and what users should do — brhydon · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27