2026-09-01
A Trinity survey taxonomizes multi-LLM routing and cascading into six paradigms plus a when/what/how design space, citing MixLLM at 97.25% of GPT-4 quality for 24.18% of the cost.
Pin every request to the strongest model you can afford, and easy lookups still pay for hard reasoning while hard problems still break the cheap model. Static serving sizes the fleet for the worst case and wastes the average case.
This Trinity College Dublin / ADAPT survey looks at a different lever: pick among independently trained LLMs at inference time. That is a different object from mixture-of-experts. MoE routes tokens to experts inside one network. Here the unit is a whole model, a whole API bill, a whole latency SLO.
Routing makes one assignment. Cascading tries a cheaper model first and escalates when a quality check fails. Production stacks usually glue both.
No new router is proposed. The paper sorts the literature into six paradigms, then overlays a three-axis design space.
Difficulty-aware routers score how hard a query looks, then send easy work to small models. BEST-Route uses a DeBERTa-v3-small multi-head router plus best-of-n sampling for the small model. vLLM Semantic Router uses ModernBERT to decide whether chain-of-thought is worth turning on. ICL-Router, GraphRouter, and IRT-Router try to make the model pool swappable: new models get a capability profile instead of a full router retrain.
Preference-aligned routers learn from Chatbot Arena votes or synthetic pairwise labels whether the strong model will win. RouteLLM compares four router backbones on MT-Bench; matrix factorization is the cheap competitive option. Arch-Router stuffs domain-action policies into the context of a 1.5B model so operators can edit policy without retraining, at the cost of a heavier router than a classifier.
Clustering methods such as UniRoute run K-means on unlabeled queries and score each model per cluster with a cost correction. New models only need to be evaluated on existing clusters. Avengers-Pro claims the resulting Pareto front beats GPT-5-medium; the survey reports that claim without a reproduced table.
On the RL side, Router-R1 trains 3B routers with PPO and allows up to four think/route steps per query. R2-Reasoner decomposes into subtasks, then allocates them, reporting an 84.46% API cost cut. Bandit systems (MixLLM, PILOT, GreenServ) update online from user feedback or measured GPU joules.
Uncertainty work asks when a small model's answer is trustworthy enough to keep. Probe classifiers and perplexity beat verbalized self-confidence. Chuang et al. find small models match large ones on the top 20% most confident queries. CP-Router uses conformal prediction to decide whether to escalate from a standard LLM to a verbose reasoning model such as DeepSeek-R1.
Cascades add a second chance. FrugalGPT is a router plus DistilBERT quality estimator plus a stop judge; the authors used a query-agnostic model order for simplicity. Cascade routing re-picks the next model at each step and may skip or reorder. AutoMix does few-shot self-verification and treats the escalate-or-stop choice as a POMDP.
The three-axis overlay labels each method by when it decides (before generation, after, or multi-stage), what it reads (query, model metadata, response, feedback), and how it computes (heuristic, supervised, bandit, RL). Section 10 sketches a control pipeline: cheap pre-router, post-generation verifier, then accept, refine, reject, or escalate.
This is a survey. It does not run a unified bake-off. Numbers below belong to the cited papers and are not comparable across rows.
| Source | Reported result | Baseline |
| MixLLM | 97.25% of GPT-4 quality at 24.18% of the cost | under a latency constraint |
| R2-Reasoner | 84.46% lower API cost | vs the SOTA baselines in that paper; accuracy described as competitive |
| GreenServ | +22% accuracy, -31% energy, <8 ms routing overhead | vs random routing over 16 open models |
| Chuang et al. | SLMs match LLMs on the top 20% high-confidence queries | eight UQ methods on edge devices |
The evaluation section is more useful than the scattered scores. RouterBench ships 405k precomputed outputs from 11 models on 7 tasks. RouterEval claims 200 million records over 8,500+ models. LLMRouterBench covers 21 datasets and 33 models. RouterEval's headline, repeated here, is that a competent router can beat the best single model by using complementarity.
If you already fan queries across APIs or mix a 7B with a 70B, this paper is a map. Three questions cut most of the design space: do you need to see the answer before you decide, which signals you actually have, and whether the router itself is cheaper than a wrong call.
Latency-critical paths want matrix factorization, clustering, or a small classifier, not a 1.5B policy model. Multi-step RL routers pay extra calls, so they make sense when API tokens dominate and extra latency is acceptable. If the model pool churns, prefer profile-based designs that add a model without retraining the router. Do not trust a model that says it is sure; probes need weights and labels, but they track correctness better.
None of that ships a new open-source router. It does stop people from rediscovering the same three failure modes.
The authors flag generalization (many routers break on a new model or domain), thin work on true multi-stage cascades, and an almost empty multimodal shelf. Text-only hidden-state probes smear once visual tokens appear.
Section 10 also marks structural holes. No current method pairs response-level signals with online adaptation: cascades look at the answer but freeze their thresholds, bandits adapt but only from the query. Learned escalation policies inside cascades are rare. Quality, cost, and latency are usually collapsed into a single λ rather than treated as a joint objective.
A second caveat sits with the reader. 97% quality and 84% savings come from different papers, different pools, different cost models. The survey never puts Table 2 onto one RouterBench Pareto. The design-space matrix in Table 1 is color-coded, so a text extract loses which cells are filled.