Infrathon: 10,000 GPU-hours benchmark shows agents beat human infra baselines
UW · hf · 2026-11-11
Researchers from UW, Stanford and others introduce Infrathon, a benchmark of diverse, long-horizon real-world AI infrastructure optimization tasks testing whether frontier agents can continually improve real infrastructure.
Scale: 6 models, 150 runs with 24-hour budgets, over 10,000 GPU-hours total.
Key findings
- Model rankings shift with evaluation horizon; 135 of 150 runs kept improving in the second half of their budgets, meaning short evaluations miss both gains and ranking changes
- Agents can beat human-designed baselines: Claude Opus 5.5 raised mean KV-cache compression from 6.82% (human Quant-VideoGen codec) to 19.86%
- Trajectory analysis flags task violations, unreliable testing, and premature completion as the main barriers
Code and Docker environments are open-sourced for building long-horizon agent evals.
More from Infra
- Anthropic's IPO haircut to ~$1T: open-source models squeeze from both sides — Xianbao_QIAN · 2026-11-11
- CNBC tours next-gen data center cooling: 750 gallons per minute in closed loop — SumitGup · 2026-11-11
- No, AMD won't absorb Nvidia's TSMC cuts — analyst warns of chain-reaction cancellations — SumitGup · 2026-11-11
- MLPerf at SC25: 12 university teams to run MLPerf Training in 48-hour Student Cluster Competition — TheKanter · 2026-11-11
- MLPerf Training v6.1 results land Nov 16 as MLCommons heads to SC26 in Chicago — TheKanter · 2026-11-11
- The Best Test for Clinical AI: Pull the Network Cable and See if the Visit Keeps Going — MaziyarPanahi · 2026-11-11