Routing to smaller models can backfire: costs rise 30%-89% in small harnesses

@yb2698 ran a series of experiments to test a popular community idea: using a decision model to judge each turn and route simple requests from a large model to a cheaper small one to save cost. The conclusion is conditional: with a small harness, it doesn't save money at all—costs actually rise 30%-89%; only with a large harness does routing pay off.

Confirmed

Experimental setup: the large model was Qwen3.8-27B (also written as Qwen3.5-27B in post m4), paired with vLLM d3 as the decision model, able to route some queries to Qwen3.5-9B.
Evaluation design: 10 hard tasks were sampled from TB-Lite (updated fixed version), each repeated 3 times, comparing "27B only" vs. "27B + decision routing," with harnesses covering ipython, Pi, Claude Code, etc.
Results vary by harness: for small harnesses like ipython and Pi, the overhead of introducing the decision protocol is more expensive than using the large model directly; under a large harness like Claude Code, routing is beneficial. The author concludes that delegation and orchestration suit the combination of "very large models + large harnesses."
Even measured by the number of conversation turns delegated to the small model, its share of output tokens remains very low.

Not Yet Confirmed

Follow-up directions are still being explored: trying different small models, or different routing mechanisms (e.g., multi-layer routing where the small model handles most of the work and the large model steps in as needed). The author is openly soliciting feedback; no conclusion yet.

Why It Matters

With concrete numbers, this experiment challenges the intuition that "routing to a small model always saves money," showing that the decision model's own call overhead can swallow or even exceed the savings in small-harness scenarios, offering practitioners a reference for making trade-offs based on harness size.

2026-10-23 ~ 2026-10-23 · 5 related posts

Primary sources

Testing LLM routing: decision-model delegation pays off only in big harnesses — yb2698 · 2026-10-23 00:02 (UTC+08:00)
Routing eval design: 10 TB-Lite hard tasks, 4 harnesses, 3 runs each — yb2698 · 2026-10-23 00:02 (UTC+08:00)
Small models still produce a tiny share of tokens; routing experiment seeks feedback — yb2698 · 2026-10-23 00:02 (UTC+08:00)

Routing to smaller models may cost more: harness overhead adds 30-89% — yb2698 · 2026-10-23
[source] Routing eval design: 10 TB-Lite hard tasks, 4 harnesses, 3 runs each — yb2698 · 2026-10-23
[source] Testing LLM routing: decision-model delegation pays off only in big harnesses — yb2698 · 2026-10-23
Routing experiment results: delegation only pays off in large harnesses — yb2698 · 2026-10-23
[source] Small models still produce a tiny share of tokens; routing experiment seeks feedback — yb2698 · 2026-10-23