How Inference Compute Shapes Frontier LLM Evaluation
Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec
cs.AI
2026-06-16
UK AISI raises inference budgets 1–3 orders of magnitude on seven hard benches. FrontierMath and HLE gain ~12 points; SWE barely moves. Newer models win on reach and reliability.
Harder evals now run long trajectories: tools, retries, interactive environments. Scores therefore depend on how much test-time compute the protocol allows. Many leaderboards still publish one tight budget. A low number can mean the model cannot do the task, or that the budget cut it off.
The UK AI Security Institute holds the scaffold fixed, raises token budgets by one to three orders of magnitude, adds context compaction and up to 999 submissions, and plots score against inference compute.
The main suite crosses six frontier models (Opus 4 to 4.6, GPT-5 to 5.4, May 2025 to March 2026) with five benchmarks: TerminalBench, SWE-Bench Pro, FrontierMath, HealthBench Hard, and HLE with multiple-choice items removed. Two cyber evals are reused for scaling curves only, expanding coverage to 12 models including Mythos Preview.
Three interventions are deliberately plain, meant as a reproducible lower bound rather than maximal elicitation:
Five independent trajectories per task per condition. Stateful tasks get bash and python. HLE and HealthBench are tool-less and graded by GPT-4o-mini. Non-cyber sets are capped at 100 tasks, so absolute scores are not comparable to published leaderboards.
Gains from typical published budgets to the expanded caps vary sharply:
| Benchmark | Typical budget | Cap here | Mean gain |
| FrontierMath | 1M | 10M | +11.7 pp |
| HLE | 64k | 5M | +11.9 pp |
| TerminalBench | 7.3M | 10M | +1.3 pp |
| SWE-Bench Pro | 16M | 30M | +0.3 pp |
| HealthBench | 16k | 10M | +0.3 pts |
HLE gains +15.5 with oracle feedback and +8.3 without. The two software-engineering benches already run large default budgets; stretching them further buys almost nothing. HealthBench barely moves even from a tiny typical budget. TerminalBench, CTFs, and The Last Ones still rise at the tested caps for most models.
Later generations mainly unlock harder tasks and solve already-unlocked ones more consistently. Token efficiency improves unevenly: clearer on CTFs and HealthBench, not significant on FrontierMath or HLE. Reach rises with generation on all six decomposed benches, most on FrontierMath and CTFs.
Repeated submission helps everywhere, from +6.4 points on FrontierMath to +17.3 on HLE (1.11× to 1.71× over the first attempt). On HLE, iteration adds +25.2 under oracle feedback versus +9.3 without. FrontierMath and HealthBench realize 90% of the iteration gain within three submissions; HLE needs 12–13. Splitting the same total budget into ten shallow trajectories lifts HealthBench from 0.501 to 0.784 and HLE from 0.417 to 0.606; stateful engineering and math move only about 0.03. Newer models extract less from parallelism; the latest ones even lose on FrontierMath when going from pass@1 to pass@10.
A score is a function of protocol. Safety and trend tracking that pin one fixed budget will miss capabilities that only show up when newer models get room to search. The paper's advice is concrete: report curves over compute, treat retries, feedback, and width-versus-depth as part of the measurement, and compare generations at matched budgets over a shared range.
For practitioners: extra inference still pays on HLE, contest math, and cyber tasks that admit search and verification. Clinical QA and SWE evals that already use large budgets return little. Newer models get more from one deep trajectory; spraying pass@k is not automatically cheaper.
Only one ReAct-style scaffold is tested. The near-flat HealthBench curve could be a knowledge ceiling, a saturated judge, or a scaffold mismatch. The protocol omits adaptive branching and verifier-guided search, so the numbers are a lower bound. FrontierMath uses 12 items, so item-level noise is large, and newer models may have seen the public set. The repetition guard is an LLM judge of semantic equivalence, and its strictness likely differs by domain. The cyber datasets sit outside the fully crossed design, so they cannot speak to feedback effects.