Cognition & OpenRouter on Model Routing: Why Naive Task Routing Fails for Agents

AI Engineer · youtube · 2026-08-07

The Pitfalls of Model Routing

Small models out of distribution: Alex Atallah (OpenRouter) notes that pushing small models beyond their training distribution causes them to thrash (e.g., calling tools in loops), ultimately costing more than just using a frontier model.
Static routing is fragile: Walden Yan (Cognition) argues that as an agent session evolves (e.g., from a codebase query to live debugging), the model chosen at the start can easily become stranded.
Cognition's approach: Keep a frontier model for high-level planning and delegate implementation to cheaper models. This cut costs by 40% while allowing parallel exploration of codebases via sub-agents.

Context & Cache Optimization

Keeping KV Cache warm: Cognition favors a single sidekick with continuous running context over multiple sub-agents, reducing cached token costs by roughly tenfold.
The hidden cost of compaction: Yan emphasizes that context compaction should be done for intelligence, not just cost. Compaction forces a cache miss, and model quality falls off a cliff well before the advertised million-token window.

More from coding & agent