Terminal-Bench 4.0 Out: Calibrated Resources, Saturated Tasks Removed, Continuous Releases
cwolferesearch · x · 2026-08-30
Terminal-Bench has released version 4.0 (only a month after 3.0), marking a transition to continuous-style benchmark updates. The release calibrates task resources, fixes tasks, and removes saturated tasks:
- Resource calibration: following the methodology from "Quantifying infrastructure noise in agentic coding evals", tasks were run with large resource headroom to pick reasonable settings; all tasks now use a flat 8-hour agent timeout, so frontier models rarely hit timeouts. Agent timeouts and errors drop significantly vs 3.0; remaining errors are mostly model refusals and output-token-exceeded.
- Sonnet 5 issues: Sonnet 5 sometimes hits timeouts and output-token limits — it uses enormous token counts (21.6B on its leaderboard run vs 6.5B for Opus 5) and shows large variance in execution time; enabling the 128k max-output-tokens setting may help.
- Task cleanup: 8 tasks removed (2 saturated, 2 refusals, 2 with public solutions, 2 quality/platform issues); a task counts as saturated when all families of the latest model generation solve it 5/5. 19 tasks were fixed for instructions, environments, and verifiers.
Related event: Terminal-Bench 4.0 Released: Opus 5 Leads, GLM 5.3 Top Open-Source Model(14 posts)→
More from coding & agent
- Vercel Goes Full Backend: Python GA, Containers, and Daemons — cramforce · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Vercel Ship Demos Container Deploys for Go, Ruby, Java and More — edwin · 2026-10-16
- TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
- Databricks engineers land impactful PRs on day 2 with AI dev tools, vs week 3 before — Yuchenj_UW · 2026-10-16
- Perplexity open-sources SPACE, the sandbox platform powering its Computer agents — AravSrinivas · 2026-10-16