FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing
Wang Wei, Harry Yang, Tiankai Yang, Samyadeep Basu, Hongjie Chen, Andy Zhao, Franck Dernoncourt, Ryan A. Rossi, Hoda Eldardiry
COLM 2026
cs.LG, cs.AI
2026-09-30
FlexRouter recasts LLM routing as DPP-based subset selection that directly optimizes answer coverage, reaching average Success@10 of 0.863 and 0.991 on RouterEval and beating independent-scoring and bandit baselines.
LLM routing usually works by scoring every candidate model independently and taking the top k. Two things get lost. Scores are computed per model, so correlations between models never enter the decision. Models with similar training data and architectures tend to fail together, and independent top-k selection happily picks several near-duplicates that produce nearly identical outputs, wrong in the same way. And k is a fixed budget, so easy and hard queries cost the same.
The paper targets a pipeline that is increasingly common in practice: multiple candidate answers are generated, then a downstream verifier, reranker, or human picks the final one. What matters there is answer coverage, the probability that at least one selected model gets it right. If the whole subset fails, no downstream selector can save you. Independent scoring is systematically suboptimal under this objective.
FlexRouter casts routing as coverage-oriented subset selection and parameterizes the policy with a DPP (Determinantal Point Process), which defines a distribution over subsets that favors high-quality, dissimilar combinations. Three pieces:
One design choice worth flagging: similarity comes from learned model embeddings, not from semantic similarity of actual outputs. The latter is left as future work.
Evaluated on RouterEval with shared candidate pools: 3,811 models (medium) and 5,000 models (large). Metric is Success@10, the fraction of queries where at least one selected model is correct.
| Method | Medium-pool avg | Large-pool avg |
| Ref. (strong single model) | 0.619 | 0.855 |
| EmbedLLM | 0.8329 | 0.9802 |
| BaRP | 0.8320 | 0.9142 |
| FlexRouter | 0.8632 | 0.9914 |
On the medium pool the biggest gain is GPQA (0.8487 vs 0.7731 for EmbedLLM); EmbedLLM actually wins MATH (0.8264 vs 0.8075), which the authors attribute to a small set of specialist models where independent high-score selection already works. Out-of-domain results are best in both settings (TruthfulQA 0.9951, WinoGrande 0.9976), suggesting the policy learns general complementarity rather than task-specific preferences.
On redundancy, measured by ILD@10 (average pairwise cosine distance among selected models in a fixed pretrained embedding space), the gap is wider: medium-pool average 0.7620 for FlexRouter vs 0.2112 for EmbedLLM and 0.1982 for BaRP. A kernel ablation favors cosine over RBF for diversity at competitive success rates. Around τ≈0.2, average subset size shrinks substantially while Success@10 holds nearly flat.
If you run multi-model orchestration or agent backends with a verifier or reranker downstream, this is a directly borrowable idea: the routing-stage objective should be coverage, not average correctness, and DPPs turn that objective into something trainable. The stopping threshold is a coverage–cost knob you can turn at inference without retraining. The stack is light: a RoBERTa-base encoder plus greedy selection.
It is a targeted fix, not a rewrite. The gains presuppose a pipeline that keeps multiple candidates and picks among them; if your system returns a single answer with no selection stage, coverage is not the right objective and none of this applies.
The authors list three, all fair: Success@k is a routing-stage coverage metric, not deployed accuracy, and end-to-end evaluation with a concrete downstream selector is future work; diversity is measured in a fixed model-embedding space, which does not guarantee output-level semantic diversity (RouterEval lacks full generations); cost is proxied by number of invoked models, ignoring per-model latency, token pricing, output length, and batching.
Two more concerns from reading the tables. On the large pool every method lands above 0.98. With 10 picks out of 5,000 models, hitting at least one correct answer is close to guaranteed, so that setting barely discriminates between methods; the medium-pool numbers carry the argument. And ILD is computed in a fixed embedding space of the same kind the method itself learns, so the measurement is not fully independent of the thing being measured; treat the absolute diversity numbers with some discount.