Notion details its Auto model router: Elo-based evals, 62-93% cheaper than flagship models
ivanhzhao · x · 2026-10-21
Notion engineer ivanhao shared a blog post on how the company built one of the most efficient knowledge work agents, centered on its upgraded model router Auto.
Key points:
- Notion brought back Elo ratings for model evaluation: real-life knowledge work is messy, and relying on a single LLM to judge another model's output doesn't reliably capture that complexity.
- A quality-vs-cost chart over 20+ common use cases shows Auto achieving near-flagship quality at the lowest credits per task — using 62% fewer credits than Sol 6.1, 87% fewer than Opus 5.5, and 93% fewer than Astra.
- Auto dynamically routes each task to the right model, with rigorous evals driving continuous improvement.
More from coding & agent
- 90% of big-company knowledge lives in heads, and "company brain" startups are in for pain — gabriel1 · 2026-10-21
- EAU: Co-evolving user simulator, environment, and RL setup for training LLM agents — UAlberta · 2026-10-21
- Meta's Self-Credit: Dense Rewards from Sparse RLVR via Intra-group Rescaling — meta · 2026-10-21
- Dev sets up automation so agents proactively pick up stale issues and PRs — vikvang1 · 2026-10-21
- Tamper-proof agent logs still can't prove the agent did the job: who reads the record back? — ZealousidealAct9661 · 2026-10-21
- One prompt: Claude builds an animated mech spider in Blender via MCP, first try — sidahuj · 2026-10-21