Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing
Guannan Lai, Han-Jia Ye
cs.AI
2026-09-29
RouteFM pretrains a single frozen LLM router across heterogeneous environments; on unseen MMR-Bench it beats the strongest baseline by 2.23 points from 8 observations per model.
LLM routing assigns each incoming query to the most suitable model in a heterogeneous pool, trading answer quality against inference cost. Training a router for one fixed pool is a solved problem; keeping it useful is not. Standard practice fits a router to one query workload and one candidate pool, so whenever a new model ships or the domain shifts, operators must collect fresh query-model performance data and retrain. The paper calls this pattern "local fitting" and targets its core defect: the router never becomes reusable infrastructure.
RouteFM makes one structural bet: the router knows no model by name. Every candidate is described solely by behavioral context, a set of (query embedding, observed quality, relative cost) triples, with model names and parameter counts withheld. Differences between candidates can only come from observed behavior, which is exactly the information that transfers across environments.
Three components:
Pretraining is episodic, drawing from four sources (212,470 query-task instances, 1.73M query-model observations) with varying tasks, pool composition (6-24 candidates), candidate order, and context budgets from K=8 to 1024. The loss combines point-wise regression, pairwise ranking, and a routing-regret term whose weight ramps from 0 to 0.25 over a 10,000-step curriculum. The whole router is 12.78M parameters.
The design is coherent: anonymity forces the model to learn capability inference from behavior, permutation equivariance blocks slot memorization, and the regret term aligns accurate prediction with good decisions.
Primary metric is routed quality (average quality of the selected model), with matched pools and observation budgets:
| Setting | RouteFM | Best baseline | Gap |
| RouterEval in-domain, K=8 | 0.6192 | 0.6108 (Embedding-Ridge) | +0.84 |
| RouterEval in-domain, K=64 | 0.6611 | 0.6601 (EmbedLLM) | +0.10 |
| MMR-Bench cross-modal, K=8 | 0.7323 | 0.7100 (EmbedLLM) | +2.23 |
| MMR-Bench cross-modal, large budget | 0.7614 | 0.7607 (UniRoute) | +0.07 |
MMR-Bench is a multimodal routing benchmark (10,370 queries, 9 candidate models) excluded from pretraining. The pattern is consistent: the fewer the observations, the larger the margin; with abundant evidence the methods converge.
The deployment-side numbers matter more. A new model enters the pool and, from 8 observations, frozen RouteFM already beats every refitted baseline; at 16 observations, routing over the expanded pool beats routing over the original one. The frozen router beats the same architecture trained from scratch on 20% labeled target-domain data by 1.08 points at K=8, while target-domain fine-tuning gives mixed, marginal changes. Context efficiency: 1% of the available observations matches Weighted kNN at full context, 3% matches MLP, 26% matches EmbedLLM.
Ablations: removing the candidate-pool Transformer costs 1.33 points at K=8, the largest single drop, confirming joint comparison beats independent estimation. Swapping the Qwen3-VL encoder for a text-only BGE encoder also degrades cross-modal transfer.
For teams running multi-model inference, router maintenance turns from a retraining problem into a context-construction problem. A new model needs a dozen probe queries, not a training run. The gains concentrate exactly where deployment hurts: cold pools and thin evidence. With rich observations the margin over simpler methods nearly vanishes, so for warm, stable pools this is incremental; the value sits in cold and dynamic ones.
The authors' own list: pretraining covers only four sources, so scaling behavior is untested; cost is normalized per episode and is not an absolute dollar or latency signal; candidates are described by behavior only, without system-level signals like latency or availability; continually evolving pools remain future work.
A close read raises more. Knew = 0 and 1 are cold-start stress tests, which the authors explicitly say should not be read as reliable zero-shot incorporation. The 0.07-point lead in the large-budget regime is effectively a tie. In-domain gains run 0.10 to 0.84 points, modest. ICL-Router uses text-only queries on MMR-Bench while RouteFM uses a multimodal encoder, so that comparison is not fully even. Simple non-parametric methods remain faster in absolute latency. All pretraining supervision comes from benchmark scores, so label noise in real deployments is untested.