Terminal-Bench 4.0 Sets Flat 8-Hour Agent Timeout After Lab Timeout Discrepancies Skewed Scores
langstonnashold · x · 2026-09-17
Terminal-Bench 4.0 has standardized all tasks to a flat 8-hour agent timeout, noting that frontier models now rarely or never hit the limit. The change addresses long-standing inconsistency in eval methodology.
Langston Nashold of Vals said the fix was sorely needed: their team fielded many questions about why their tbench numbers didn't match lab-published results, and in 90% of cases the answer was that Vals used standard timeout limits while labs had raised their own. Differing timeout settings can directly reshuffle agent leaderboard rankings.
Related event: Terminal-Bench 4.0 Unifies 8-Hour Timeout to Curb Benchmark Gaming(2 posts)→
More from Models
- Google open-sources EmbeddingGemma 2: on-device multimodal search across photos, audio, docs — bigaiguy · 2026-11-02
- Anthropic's server-side rule reignites the case for owning open models over renting AI — alex_verem · 2026-11-02
- Epoch AI: open-weight GLM 5.3 exploited the patch gap to hack 6 Firefox releases at ~$1,000 each — rohanpaul_ai · 2026-11-02
- SemiAnalysis: safety classifiers sandbag scores of Opus 5.5 and GPT-6 Astra — morqon · 2026-11-02
- Liquid AI's 1.7B LFM3 becomes first local model to nail live fixtures lookup on a MacBook — NielsRogge · 2026-11-02
- Claude upgrade-page traffic rebounded to 17.6M in September, near May's 18.6M peak — firstadopter · 2026-11-02