Anthropic: same 600K tokens, an advising agent scores 89 vs 76 for pure execution
AI Engineer · youtube · 2026-09-14
In an AI Engineer talk, Anthropic's Katelyn Lesse and Angela Jiang challenge the assumption that tokens are fungible and budget is the only lever. On a fixed 600K-token financial-analysis bench: a pure executor scored 76, while an agent spending part of the identical budget consulting an adviser scored 89. They present three strategy primitives — advising (executor calls out mid-task), grading (rubric up front, a grader sends the executor back to iterate), and dreaming (reading the executor's own transcripts into memory for the next run). The sharper finding: rescored the way a real analyst would — an 80%-accurate P&L is useless, so only perfect runs count — the execute baseline passes 42% of the time, meaning 3 runs and 1.8M tokens of true cost per clean answer. Advise and grade reach the same bar for considerably less; which strategy wins depends on whether you optimize token efficiency or single-run reliability.
More from coding & agent
- Gemini 4 tops public coding benchmarks but drops to bottom on private tasks — Hesamation · 2026-10-30
- User builds 20-bot Grok suite to track wins, prep 1:1s and close promises — NicoVerderosa · 2026-10-30
- Why 30-year-old BM25 beats vector embeddings for agentic search, per search vet Jo Kristian Bergum — jobergum · 2026-10-30
- LangChain CEO opens Interrupt London with keynote on why orgs should own their intelligence — LangChain · 2026-10-30
- Simlock: open-source control plane gives parallel coding agents their own iOS/Android simulators — Scobleizer · 2026-10-30
- Keyfleet launches open canvas where personal AI agents team up with a shared USDC treasury — seanwbren · 2026-10-30