AI evals are the new compute bottleneck: GAIA run costs $2,829, HAL spent $40K
evijit · x · 2026-10-19
EvalEval Coalition argues AI evaluation costs now gate who can do research: HAL spent $40,000 on 21,730 agent rollouts (9 models × 9 benchmarks); a single GAIA run on a frontier model costs up to $2,829 pre-caching; Exgentic's $22K sweep found a 33× cost spread across agent scaffolds on identical tasks. HELM's aggregate cost back in 2022 was already $100K. New agent benchmarks are noisy and scaffold-sensitive, hard to compress, and reliability via repeated runs multiplies costs further.
Related event: AI Benchmarks Face Saturation and Soaring Evaluation Costs(3 posts)→
More from AGI Musings
- Stanford's Noah Goodman Shares First-Pass AI Safety Concept Map for Lecture — noahdgoodman · 2026-10-19
- Agents may become the best distribution channel ever, argues VC on platform shifts — omooretweets · 2026-10-19
- Two Years On, the General LLM Bet Has Paid Off With No Wall in Sight — adivinemessenger · 2026-10-19
- Amazon blocks agentic commerce, JP Morgan blocks agentic banking — one agent will win — krishnan · 2026-10-19
- Kenney warns against AI slop; dev counters: AI is a tool for makers, not a substitute — majidmanzarpour · 2026-10-19
- Harvard's Boaz Barak opens AI safety course mocking OpenAI conflict of interest — zainhas · 2026-10-19