New Long-Horizon Terminal Benchmark Released
minchoi · x · 2026-07-14
The newly released Long-Horizon Terminal-Bench is designed to test long-horizon terminal task capabilities, rather than short tasks that can be completed in a few minutes.
- The benchmark includes 46 real-world terminal tasks across 9 categories
- Evaluated 18 frontier models using the unified Terminus-2 harness
- A single task can last up to 90 minutes and involve roughly 120–320 steps
- Employs a hidden, replay-based verifier to prevent shortcuts and reward hacking
- Introduces continuous partial scoring to avoid compressing "progress made" into a simple 0/1
The author concludes that:
- The best model only achieves an average reward of 0.505
- 29/46 tasks remain unsolved by any model
- Long-horizon agents can currently "start working," but are still far from "getting the job done"
The post also mentions that Grok 4.5 currently tops the leaderboard, but overall, current models are still far from achieving truly reliable terminal execution.
Related event: Tencent Hunyuan Releases Long-Horizon Terminal Bench Exposing Model Limits(6 posts)→
More from Models
- MissionReport benchmark: only 3 of 540 AI-generated government reports pass all criteria — Exp_Mark · 2026-10-27
- Local beats cloud: Qwen 3.8 FN handles private student data as well as frontier models — dr_lm · 2026-10-27
- Aidan Gomez: model's CoT said no tool calls right before making them — aidangomez · 2026-10-27
- Nat Lambert: Cohere's model is narrow RL training on internal infra ill-suited for open weights — natolambert · 2026-10-27
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27