METR: Serious Evals Need Massive Token Budgets
idavidrein · x · 2026-07-03
David Rein recommended an AISI (AI Safety Institute) blog post, pointing out that if you are running model evaluations, you are likely not using enough tokens.
Using METR as an example, he noted that their hardest tasks already consume 500 million to 1 billion tokens (including cache); otherwise, newer models don't have enough room to perform. This highlights the necessity of a sufficient inference budget for serious agent evals.
More from coding & agent
- General agents burn ~140M tokens/month on LinkedIn outreach; scripts do it 60x cheaper — ChrisHarpon2 · 2026-10-26
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- LangChain ships Managed Deep Agents 0.9, walkthrough from Interrupt London keynote — LangChain · 2026-10-26
- Knowledge Worker Kit adds a full section on organizing projects — dSebastien · 2026-10-26
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- METR red-teams OpenAI's agent monitoring, finds bypasses and gaps — thione · 2026-10-26