LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai, Liangqi Yuan, Zijie Lei, Weizhi Zhang, Kunlun Zhu, Haodong Yue, Keyang Xuan, Ge Liu, Jiaxuan You
cs.CL
2026-08-07
LLMRouter unifies LLM routing as a cost-aware decision process; across 18 models and 16+ routers, learned routing beats always-picking-the-largest by 14.6%.
LLM routing solves a practical problem: no single model is optimal on every query. An expensive model can still get some questions wrong while a cheaper one is good enough elsewhere. A router picks which model should answer each query, cutting cost, matching queries to the model best at them, and adapting to user preferences.
Plenty of methods already exist: binary routers that choose between a weak and a strong model, cost-aware cascades, graph-based routers that pass messages over a query-model graph, and agentic routers trained with reinforcement learning. The trouble is that they live in different formalisms, incompatible codebases, separate supervision, and different candidate pools. Fair comparison becomes impossible, and you cannot tell whether a performance gap comes from the router or the experimental stack around it. Evaluation is even harder: scoring a router means running every candidate on every query with task-specific metrics, and existing benchmarks only cover single-turn text, leaving multi-turn and personalized routing without a unified, cost-aware protocol.
The authors cast all routing as one cost-aware sequential decision process. At each step the router observes a state (query, optional user context, accumulated history) and takes an action: dispatch the state to some candidate model and append its response, or terminate and aggregate a final answer. The objective is to maximize answer quality minus inference cost.
Under this formulation a router is described by five components: a context encoder that turns the state into a decision basis, a model encoder that characterizes each candidate, a scoring function and decision rule that convert state and candidate representations into a routing action, and a learning signal that fits these components toward the optimal policy. Existing routers fall into three families by what state they observe: single-turn (query only), multi-turn including agentic (query plus history), and personalized (plus user context). Swapping a router is a configuration change, not a rewrite.
Around this formulation the team built three things. First, the open-source LLMRouter library: six modules, where adding a router means subclassing one base class and implementing a routing method plus a loss function. It ships more than 16 routers and can expose any of them as an OpenAI-compatible server, deployable to Slack or Discord via OpenClaw, or prototyped by dragging nodes in ComfyUI.
Second, the library automates supervision construction: sample queries from existing benchmarks into a unified format, split train and test, dispatch each query to 18 candidate models (7B to 671B open weights, via Together and NVIDIA NIM), collect responses with token counts, score by task metric, and price by tokens. The output is a dense query-model matrix of performance and cost that serves as both training supervision and test bed. Adding a task takes one prompt template and one metric.
Third, this pipeline yields xRouteBench: five tracks and 4,767 instances covering generic tasks (MMLU, GSM8K, HumanEval, and others), memory, vision (image and video), time series, and personalization.
Under the performance-first setting, several findings hold firm.
| Finding | Data |
| No single router dominates | GraphRouter has the best average (45.46) but is not consistently first on any task |
| Learned routing beats always-picking-the-largest | 14.6% relative gain; the largest model costs the most yet performs only middling |
| Multi-turn does not reliably beat single-turn | Router-R1 averages 22.30, dead last; multi-turn depends on the base model (Qwen2.5-3B) |
| Personalization pays, but modeling matters | Under the persona judge GMTRouter 68.78 > PersonalizedRouter 67.86 > best user-agnostic EloRouter 66.40 |
Real users make the point sharper: on preferences collected live through Slack, the personalized router (83.05) and EloRouter (82.20) agreed most often with users. Rankings also flip as cost tightens: RouterDC tops generic tasks when only quality matters (80.56) but falls to 10th of 11 under the heaviest cost weighting, while MLPRouter sits last on vision in the quality-first setting yet overtakes everyone from cost weight 0.4 onward.
For teams shipping multiple models, this paper straightens out a field that talked past itself: one unified abstraction, one reusable training and evaluation pipeline, one deployable library. Two conclusions are immediately practical. First, do not reflexively call the largest model; it is the priciest and not necessarily the best, and a lightweight learned router buys a better quality-cost trade-off. Second, which router to pick depends on the cost budget, and the paper spells out who wins at each performance-cost operating point.
The candidate pool is open-weight only; the most expensive proprietary frontier models are absent, which affects how both cost and the capability ceiling are characterized. The multi-turn verdict is constrained by the base model: decomposition and aggregation uniformly use Qwen2.5-3B, and a stronger base could change the outcome. The personalization track relies on a persona judge rather than fully human-annotated labels; live Slack preferences back it up but the sample is small (15 users, 40 sessions). And routing only decides who answers; it does not touch inference optimization inside a single model.