Benchmark cheating is rampant: researcher flags Terminal-Bench 4.0 gaming problem
xeophon · x · 2026-09-08
Researcher xeophon demonstrated cheating in Terminal-Bench 4.0, stressing the point isn't that tb4.0 is a bad eval, but that gaming is rampant across modern benchmarks.
Possible fixes: using harbor analyze plus third-party regrading of flagged cheats to 0 — but getting third parties to adopt this is hard. A deeper open problem: prompts defining exactly what counts as cheating vs. a valid solution risk spoiling the answer if written too precisely.
More from Models
- Artificial Analysis breaks down all sub-evaluations in Intelligence Index v4.3.2 — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra gains ~20 points on Terminal-Bench 4.0, rivalling GPT-6.1 Sol — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra cuts hallucination rate by 50 points while lifting accuracy 2 points — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27
- Leaked Terminal-Bench 4.0 chart shows GPT-6.1 Terra leading at lowest cost, unverified — Angaisb_ · 2026-10-27
- OpenAI brings GPT-6.1 Terra to the API and Codex for fast, cheap computer use — OpenAIDevs · 2026-10-27