Whose turn is it? Context-guided model routing for stateful agents
Anikesh Parashar, Avik Dutta, Vashu Chauhan, Sumit Gulwani, Priyanshu Gupta, Hosein Hasanbeig, Vu Le, Ashish Tiwari, Jack Williams
cs.AI
2026-10-14
Routers that read session history gain 3.3 pass points on multi-turn spreadsheet agents at matched offload; cache math says switching down pays back in ~1 call, switching up never does.
Model routing sends easy requests to a cheap model and hard ones to an expensive model, cutting cost without hurting quality. Almost every existing router treats each request as an independent input. Deployed agents break that assumption: one user turn can contain dozens of model and tool calls, the difficulty of the current request depends on state built by earlier turns ("apply the same format to Q3" is only interpretable if you know what was built), and switching models mid-session forfeits the prompt cache and re-bills the conversation at uncached rates.
The Microsoft team ran the study on spreadsheet manipulation, a domain that is unusually hard on routers: workbooks are persistent mutable state, formulas create hidden dependencies through cell references, there is no test suite or compiler to check against (a formula over the wrong range returns a plausible number, not an error), and end-user requests are short and underspecified. Models are far from saturating it: Opus 4.8 passes at most 31.2% of multi-turn SheetCopilot, and the best of eight frontier models completes 34.9% of SpreadsheetBench 2 workflows, with debugging accuracy as low as 12%.
Routing happens at the user-turn boundary: one decision per user request, and the selected model owns every LLM call and tool interaction of that turn. This granularity sits between committing one model per session and re-deciding on every call. Handoffs pass only serialized dialogue, tool observations, and workbook state; no KV cache is shared.
Three dimensions vary systematically:
Zero-shot-only is a practical constraint: supervising a multi-turn router requires labels, and since state depends on every earlier choice, a T-turn session admits up to 2^T routed trajectories. Cost accounting charges the router's own calls and cache effects to every policy. Benchmarks are multi-turn conversions: SheetCopilot keeps 112 multi-subtask tasks averaging 2.88 turns; SpreadsheetBench 2 becomes 221 workflows averaging 4.76 turns.
The central comparison holds the router fixed and varies only the history input. Offload shares are nearly identical, yet pass rates differ by 3.3 points.
| Policy (SheetCopilot) | Pass rate | Turns on cheap model |
| Expensive-only | 27.7% | 0% |
| Cheap-only | 26.8% | 100% |
| Router, no history | 25.0% | 23.3% |
| Router, with history | 28.3% | 22.7% |
The gain comes entirely from which turns get offloaded, and it is positive for every pair: +5.4 for Sol/Haiku, +2.7 for Sol/Luna, +1.7 for Opus high/low. The history-aware router stays within one point of expensive-only. Without history, follow-ups such as parameter changes or references to earlier objects look deceptively simple and get misrouted. On SSB-2 both routers barely offload (0.4–1.7% with history) and the gap shrinks to 29.3% vs 28.0%.
Rules without a signal fail across the board. Random routing offloads about half the turns and lands below both fixed endpoints for every pair. Escalation keeps 90%+ of turns on the cheap model and lifts the widest-gap pair (Sol/Haiku) from 20.5% to 24.1%, but on the Opus pair it costs $2.456 per task against $1.081 for cheap-only.
Two findings on cost:
For anyone working on agent cost, this reframes routing from per-request classification into a sequential decision problem, with three portable lessons. The router must read the trajectory; a history-blind router is barely better than not routing. Pair selection matters as much as the rule: when the cheap candidate already matches the expensive one (Sol/Luna, Opus at low effort), just using it dominates every routed policy of that pair, as Luna's 28.6% pass at $0.061 per task beats the router's 26.8% at higher cost. And switching should be sticky: go down early and stay there, go up only when the quality gain clearly beats a switching cost that grows with the session.
The paper's nature should be stated plainly: no new model or training method, just systematic experiments and a reusable cache formula. Incremental, but solid measurement work.
The authors' own list: zero-shot, cost-aware policies only; sessions average under three turns with a maximum of seven, so long trajectories, independent tasks, and cross-sheet interactions are untested; decomposing annotated subtasks into turns approximates rather than reproduces natural dialogue.
Two harder problems show up in the numbers. First, the savings never materialize: on the Opus pair, the only one with exact routed costs, the router costs $0.369 per task versus $0.365 for expensive-only ($2.080 vs $2.009 on SSB-2); the offload rate is too conservative and cache losses eat the price gap. Second, the router is blind to the pair: all three pairs, with very different capability gaps, get 21–24% offload, so it responds to perceived turn difficulty rather than the real gap between candidates, and a pair-aware router could offload far more where the cheap model is already as accurate. SheetCopilot follow-ups also tend to be direct continuations of the first request, which may inflate the value of history on this dataset.