Frontier Models See Massive Capability Leaps with Relaxed Compute Limits
koltregaskes · x · 2026-07-03
After the UK's AI Safety Institute (AISI) increased its evaluation compute budget from 2.5 million to 50 million tokens, a frontier model's task time horizon jumped from 40 minutes to 4 hours. Testing frontier models on cybersecurity, software engineering, and math benchmarks with significantly more compute than standard evaluations revealed continuous performance gains—about 8% of tasks were only solved with over 10 million tokens, and newer models proved much better at utilizing extra compute.
At higher budgets, the doubling rate of frontier capabilities is about 60% steeper than in standard evaluations. Standard assessments systematically underestimate true model capabilities because they restrict compute before a plateau is reached, causing the longest and hardest tasks to be prematurely cut off. These findings offer crucial insights for future model evaluation methodologies.
Related event: Frontier AI Task Duration Doubles, Slowed by Budget Constraints(3 posts)→
More from Infra
- AW-0 runs at 230ms on a single RTX 5090, 2.4x faster than autoregressive code planning — trevordarrell · 2026-10-27
- Site tracks US city council topics: data centers rank only behind pickleball courts — zakkohane · 2026-10-27
- MiniMax H3 runs fully on a Galaxy S26: 5-second video with audio in ~7 minutes — BidFunny4862 · 2026-10-27
- Unconventional AI acquires inference chip firm NeuReality in 1,000x efficiency bid — NaveenGRao · 2026-10-27
- Musk: robot RAM demand will dwarf everything else, and 100% of superintelligence compute goes to deep space — elonmusk · 2026-10-27
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27