UK AISI paper: fixed-budget evals increasingly understate frontier LLM capability
evijit · x · 2026-09-23
The UK AI Security Institute (AISI), working with the EvalEval team, released the paper "How Inference Compute Shapes Frontier LLM Evaluation," publishing methods, transcripts, and all data via Eval Cards and Hugging Face.
Key findings:
- Evals are shifting to harder, long-horizon tasks with tool use, making scores highly sensitive to test-time compute
- Up to 12 frontier models were tested on 7 challenging benchmarks (software engineering, math, medicine, cybersecurity — including FrontierMath, Humanity's Last Exam, TerminalBench) under three inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts
- Larger token budgets substantially improve scores across domains; fixed-budget evals increasingly understate frontier capability as newer models unlock harder tasks at high budgets
- Repeated submission broadly helps, but the value of larger budgets varies by benchmark
Implication: low scores may reflect eval setup rather than model capability, and compute budgets should be reported with results.
Related event: UK AISI Partners with EvalEval to Open Up AI Evaluations via Eval Cards(5 posts)→
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12