PISA-E picks LLM ensemble size and makeup per prompt, cutting cost 36% while outscoring rivals

Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini

cs.AI

2026-11-07

PISA-E predicts per-prompt LLM accuracy, then picks ensemble makeup and size with a tunable cost knob, beating top ensembles by 2.87 points or matching them at 36% lower cost.

What problem this solves

Every LLM-backed service lives with the same tension: strong models answer well but burn money, cheap models save money and occasionally fail. Routing picks one model per prompt and gives up the complementary strengths of the pool. Ensembling calls several models and aggregates, which is steadier and multiplies the cost. Existing ensemble methods also freeze composition and size at training or configuration time, so neither adapts to the incoming prompt or to a budget that shifts after deployment. The paper folds both decisions together: which models to invoke for this request, and how many, decided at inference from the prompt and an adjustable cost-quality preference.

Method

The core idea is to train one thing well, the probability that each model answers each prompt correctly, and derive every ensemble decision from that estimate. Nothing ensemble-specific is ever trained.

Results

Four benchmarks (LLMRouterBench Track-2, RouterBench, FusionBench, SPROUT), ID and OOD conditions, five seeds:

ComparisonMetricPISA / PISA-EBest baseline
Single-model, ID / OODmacro hypervolume37.41 / 22.85EmbedLLM 35.78 / 21.33
Ensemble, α=1, IDmacro task perf.75.27%SelectLLM 73.18%
Ensemble, α=1, OODmacro task perf.63.24%DFPE 60.37%
Matched cost, IDperf. / cost (×10⁻⁴ USD)73.78% / 84SelectLLM 73.18% / 132
Matched cost, OODperf. / cost60.85% / 292DFPE 60.37% / 387

At the matched-cost points PISA-E is up 0.60 and 0.48 points while spending about 36% and 25% less. Sweeping α from 1 to 0.3 cuts ID cost by a factor of 21.5 with performance sliding from 75.27% to 70.75%; OOD cost drops roughly 106×, performance from 63.24% to 50.54%. Three ablations: adaptive cardinality beats the best fixed-k variant by 0.28 (ID) and 1.80 (OOD) HV; prompt-dependent ranking adds 4.96 HV over a fixed global ranking under ID; self-weighted BERTScore beats plain BERTScore by 0.92 (ID) and 1.21 (OOD) points. At α=0.3, 84% of ID queries get a single model; at α=1 the average is 6.4 models, yet 21.6% of queries still stay single-model. Ensemble size genuinely varies by prompt.

Why it matters

"How many models should this request invoke?" has been a static config decision in production. This paper turns it into a per-prompt routing decision with the knob on the inference side: one trained predictor serves both the budget tier and the quality tier, no per-tier retuning. The parts are off-the-shelf, embeddings, clustering, logistic regression, BERTScore, so there is no heavyweight training bill. It fits multi-model gateways and agent services with heterogeneous requests and shifting budgets.

Limitations

Terms

Source

Related papers

All paper explainers