Terminal-Bench task accused of contamination: only web-searching models score full marks
xeophon · x · 2026-09-07
X user xeophon flagged that the terminal-bench protein-autointerp-disulfide task has no re-grading, and the only models that solve it are ones that look up the solution online and receive full reward. Among them: GLM-5.3 and Gemini 3.8 Flash. Follow-up data shows Gemini 3.8 Flash uses web access (web search, curl, git clone, HEAD requests) more than all other models combined on tb4.0, raising doubts about how much of its benchmark performance reflects reasoning rather than retrieval.
More from Models
- Leaked Terminal-Bench 4.0 chart shows GPT-6.1 Terra leading at lowest cost, unverified — Angaisb_ · 2026-10-27
- GPT-6.1 Terra rolls out to all Plus, Pro, Enterprise and Edu users — OpenAI · 2026-10-27
- OpenAI launches GPT-6.1 Terra with 50% lower API prices than GPT-5.6 Terra — OpenAI · 2026-10-27
- In cyber, intention matters little: Chintala on model info leaks, and what users should do — brhydon · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27