METR: Serious Evals Need Massive Token Budgets
idavidrein · x · 2026-07-03
David Rein recommended an AISI (AI Safety Institute) blog post, pointing out that if you are running model evaluations, you are likely not using enough tokens.
Using METR as an example, he noted that their hardest tasks already consume 500 million to 1 billion tokens (including cache); otherwise, newer models don't have enough room to perform. This highlights the necessity of a sufficient inference budget for serious agent evals.
More from coding & agent
- BELL: Open-Source Stack Combines Browserbase, Eve, iMessage and Stripe Link for Personal Agents — jeff_weinstein · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Two engineers rebuilt a FedEx supplier's billing core in 3.5 months vs an estimated 7-8 — alex_verem · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27
- Shift-1 router cuts coding agent spend up to 55% while keeping SWE-bench accuracy — ypatil125 · 2026-10-27
- Non-coder vibe-codes a Godot JRPG with Claude Code in two weeks, music and art synced — ADHDavid · 2026-10-27