Evals Shouldn't Reward Better Infra: 90% of Terminal-Bench Mismatches Came From Longer Lab Timeouts
xeophon · x · 2026-09-17
A discussion around Terminal-Bench scoring methodology is drawing attention. The Vals team revealed that in 90% of cases where their tbench numbers didn't match a lab's official results, the answer was simple: they used standard timeout limits while the lab had raised its own time limits.
xeophon argues eval numbers should never be influenced by infrastructure — wall-clock time is too punishing to teams with worse infra, and colocated sandboxes, low-latency deployments, and load all meaningfully move scores. A flat timeout standard keeps comparisons fair.
Related event: Terminal-Bench 4.0 Unifies 8-Hour Timeout to Curb Benchmark Gaming(2 posts)→
More from Models
- Open-source VoxLock tops Krisp voice isolation benchmark with 4.95% WER — Arbaz Siddiqui · 2026-11-02
- Anthropic's 85% inference margin sparks debate on open source's price-pressuring role — code_star · 2026-11-02
- User reports xAI appears to have quietly raised weekly usage limits, dropping his meter from 93% to 47% — XFreeze · 2026-11-02
- Meituan's LongCat-2.5 lands as 1.6T MoE agent model, with DeepSeek copycat speculation — teortaxesTex · 2026-11-02
- Asked what AI firms want you to stop owning, Claude said: your own compute — alex_verem · 2026-11-02
- Is Hermes Still Worth It? Codex's $200 Tier Now Overlaps Most of Its Value — Foreign_Lead_3582 · 2026-11-02