PISA-E picks LLM ensemble size and makeup per prompt, cutting cost 36% while outscoring rivals
Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference
Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini
cs.AI
2026-11-07
PISA-E predicts per-prompt LLM accuracy, then picks ensemble makeup and size with a tunable cost knob, beating top ensembles by 2.87 points or matching them at 36% lower cost.
What problem this solves
Every LLM-backed service lives with the same tension: strong models answer well but burn money, cheap models save money and occasionally fail. Routing picks one model per prompt and gives up the complementary strengths of the pool. Ensembling calls several models and aggregates, which is steadier and multiplies the cost. Existing ensemble methods also freeze composition and size at training or configuration time, so neither adapts to the incoming prompt or to a budget that shifts after deployment. The paper folds both decisions together: which models to invoke for this request, and how many, decided at inference from the prompt and an adjustable cost-quality preference.
Method
The core idea is to train one thing well, the probability that each model answers each prompt correctly, and derive every ensemble decision from that estimate. Nothing ensemble-specific is ever trained.
- Capability estimation (shared front end). Prompts are embedded with Qwen3-Embedding-8B. X-means over training embeddings yields centroids, and each prompt gets soft memberships across them (softmax over negative cosine distances, so one prompt can lean on several clusters). Every candidate model has its own L2-regularized logistic head mapping those memberships to a correctness probability; the cost term is the model's average observed invocation cost.
- Ensemble selection. The objective is utility J = α·p̂ + (1−α)·(1 − cost), where α is the cost-quality knob, adjustable at inference with no retraining. Training a quality predictor per model subset would be exponential, so PISA-E uses a surrogate: assuming conditional independence, a Poisson-binomial dynamic program computes the probability that a strict majority of the subset answers correctly. Candidates come from a greedy scheme: rank models by utility, then evaluate odd-cardinality prefixes (1, 3, 5…). Odd-only is provably safe: dropping one model from an even ensemble lowers the required majority by one vote and the cost by one call, so an even ensemble can never beat its odd subset under the surrogate.
- Response resolution. The selected models are queried in parallel and self-weighted BERTScore picks the final answer: each candidate is scored by its BERTScore similarity to the others, weighted by predicted capability, so models expected to be reliable get more say.
Results
Four benchmarks (LLMRouterBench Track-2, RouterBench, FusionBench, SPROUT), ID and OOD conditions, five seeds:
| Comparison | Metric | PISA / PISA-E | Best baseline |
| Single-model, ID / OOD | macro hypervolume | 37.41 / 22.85 | EmbedLLM 35.78 / 21.33 |
| Ensemble, α=1, ID | macro task perf. | 75.27% | SelectLLM 73.18% |
| Ensemble, α=1, OOD | macro task perf. | 63.24% | DFPE 60.37% |
| Matched cost, ID | perf. / cost (×10⁻⁴ USD) | 73.78% / 84 | SelectLLM 73.18% / 132 |
| Matched cost, OOD | perf. / cost | 60.85% / 292 | DFPE 60.37% / 387 |
At the matched-cost points PISA-E is up 0.60 and 0.48 points while spending about 36% and 25% less. Sweeping α from 1 to 0.3 cuts ID cost by a factor of 21.5 with performance sliding from 75.27% to 70.75%; OOD cost drops roughly 106×, performance from 63.24% to 50.54%. Three ablations: adaptive cardinality beats the best fixed-k variant by 0.28 (ID) and 1.80 (OOD) HV; prompt-dependent ranking adds 4.96 HV over a fixed global ranking under ID; self-weighted BERTScore beats plain BERTScore by 0.92 (ID) and 1.21 (OOD) points. At α=0.3, 84% of ID queries get a single model; at α=1 the average is 6.4 models, yet 21.6% of queries still stay single-model. Ensemble size genuinely varies by prompt.
Why it matters
"How many models should this request invoke?" has been a static config decision in production. This paper turns it into a per-prompt routing decision with the knob on the inference side: one trained predictor serves both the budget tier and the quality tier, no per-tier retuning. The parts are off-the-shelf, embeddings, clustering, logistic regression, BERTScore, so there is no heavyweight training bill. It fits multi-model gateways and agent services with heterogeneous requests and shifting budgets.
Limitations
- The strict-majority surrogate assumes correctness events are conditionally independent across models. The authors tested a Gaussian-copula variant that models dependencies and found no consistent gain over the simple version, but pools of similar models with correlated errors could stress the assumption harder than these benchmarks do.
- The surrogate optimizes majority-vote success while the actual resolver is BERTScore MBR; the selected subset is optimal for a metric that is not the final aggregator.
- On RouterBench, a fixed single model (k=1) achieves higher hypervolume than adaptive ensembling in both settings, so paying for extra models does not pay on every task distribution.
- The margins at matched cost sit around half a point; the 2-3 point wins show up at α=1, where the ID cost (280 ×10⁻⁴ USD/query) is more than twice SelectLLM's 132.
- Cost is a per-model average that ignores prompt length, which biases long-context estimates, and the greedy prefix search is not globally optimal (an exhaustive comparison lives in an appendix).
Terms
- LLM routing: a dispatch layer that picks which model handles each request.
- Ensemble cardinality: the number of models invoked in one ensemble call.
- X-means: a k-means extension that picks the number of clusters automatically.
- Poisson-binomial distribution: the distribution of a sum of independent Bernoulli trials with different success rates.
- Hypervolume (HV): the area of objective space dominated by a set of solutions; compresses a cost-performance frontier into one number.
- MBR decoding: minimum Bayes risk, picking the candidate with the highest average similarity to the other candidates.
Source
Related papers
All paper explainers