Anthropic: same 600K tokens, an advising agent scores 89 vs 76 for pure execution

AI Engineer · youtube · 2026-09-14

In an AI Engineer talk, Anthropic's Katelyn Lesse and Angela Jiang challenge the assumption that tokens are fungible and budget is the only lever. On a fixed 600K-token financial-analysis bench: a pure executor scored 76, while an agent spending part of the identical budget consulting an adviser scored 89. They present three strategy primitives — advising (executor calls out mid-task), grading (rubric up front, a grader sends the executor back to iterate), and dreaming (reading the executor's own transcripts into memory for the next run). The sharper finding: rescored the way a real analyst would — an 80%-accurate P&L is useless, so only perfect runs count — the execute baseline passes 42% of the time, meaning 3 runs and 1.8M tokens of true cost per clean answer. Advise and grade reach the same bar for considerably less; which strategy wins depends on whether you optimize token efficiency or single-run reliability.

Original post →

Anthropic: same 600K tokens, an advising agent scores 89 vs 76 for pure execution

More from coding & agent