Infrathon: AI2/UW/Stanford benchmark gives agents 24 hours to optimize AI infra
NielsRogge · x · 2026-11-14
AI2, UW and Stanford released Infrathon, a benchmark letting language agents optimize real AI infrastructure over 24-hour budgets. Across 6 models, 150 runs and 10,000+ GPU-hours, model rankings shift with the evaluation horizon and 135/150 runs kept improving in the second half — showing short evaluations miss both gains and true rankings. Claude Opus 5.5 lifted KV-cache compression from 6.82% (human-designed Quant-VideoGen codec) to 19.86%. Analysis identifies task violations, unreliable testing, and premature completion as reasons agents stall. Paper: arXiv:2611.11995.
More from coding & agent
- One diagram to teach LangChain tool calling: how AIMessage and ToolMessage pair up — ghumare64 · 2026-11-14
- Data curation RL is just SWE envs: code pretraining data increasingly consists of Claude-built, less-sloppy repos — willcb · 2026-11-14
- Solo dev builds real-time conference translation platform for 70 languages with an agentic harness — evisoft · 2026-11-14
- Open-source Blender add-on runs NVIDIA Kimodo, ARDY and MotionBricks locally for AI motion generation — bdsqlsz · 2026-11-14
- TestPrune: pruning regression tests to unblock AI-driven autonomous bug fixing — burkov · 2026-11-14
- Context beats model choice: Databricks benchmark shows 91% vs 58% for data agents — CautiousUse8597 · 2026-11-14