RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
Haiqiang Zhang, Yuanqing Lei, Wanting Li, Tao Zhang, Wenqi Jiang
cs.DB, cs.IR
2026-08-04
RAG exposes many configs, each a different quality-vs-performance trade-off. RAG-Stack finds Pareto frontiers without deploying every candidate, covering 52.5%-153.2% more quality-performance space than SOTA search methods at the same iteration budget.
A RAG system has a daunting number of knobs: retrieval index, top-k, reranking, generator model, and how the model invokes retrieval. Each configuration is a different trade-off between answer quality and serving performance, and no single one dominates. Picking a config for a deployment means searching a two-dimensional Pareto frontier.
The authors name three pain points: existing optimizers ignore cross-stage interactions or per-stage signals; multi-objective RAG optimizers look only at the algorithm space and miss the system design space (placement, batching, parallelism); and measuring performance by real deployment is costly and does not transfer across systems.
RAG-Stack has three parts and aims to find Pareto frontiers without deploying every candidate.
RAG-PE is a multi-objective Bayesian optimizer (an extension of MOBO) with sub-metric awareness: it uses stage-level diagnostics (retrieval context recall and precision, generation faithfulness) to guide exploration while still optimizing end-to-end quality and performance. Its candidate pool is heterogeneous: Sobol coverage, stage-guided candidates, DLS local refinement, and Pareto-tension crossover, arbitrated by LogNEHVI with forced channels on stagnation.
RAG-IR is a system-agnostic workload abstraction: a workflow schema (an order-free summary of each stage's performance attributes) plus a per-request trace (execution DAG, stage invocations, token counts), bridging quality evaluation and the cost model.
RAG-CM is an ML-analytical cost model in four layers: an algorithm layer modeling RAG stages (IVF-Flat, PQ, Fastscan, HNSW) into hardware-agnostic operator work profiles; a performance layer mapping profiles to time via roofline and Amdahl's law; a communication layer pricing data movement by topology; and an assembly layer sweeping the system design space into throughput and latency. A frontier-transfer mode reuses old quality measurements, re-scores deployments under new resources, and runs a few polishing iterations instead of searching from scratch.
On two datasets (RAGEval, 100 queries; MS MARCO, 100 queries from 8.84M passages) and two hardware setups (SysA with 4 H100s; SysB with 8 A100 80GB), against baselines GP+LogNEHVI, SMAC3, and Greedy-Forward/Lookback: at the same iteration budget, RAG-Stack's frontiers cover 52.5% (RAGEval) and 153.2% (MS MARCO) more normalized quality-performance space than SOTA search methods. Transferring a frontier to a new system covers 182.2% more space than re-optimizing from scratch. End-to-end search takes 4.59 hours on RAGEval and 3.60 on MS MARCO.
For RAG eval and deployment engineers, this turns "pick a RAG config" from manual guesswork into systematic frontier search, and RAG-CM makes results portable across hardware so you do not re-benchmark every new machine. The sub-metric awareness is practical: it looks at per-stage diagnostics, not just the end-to-end score, so you learn whether the bottleneck is retrieval or generation.
Evaluation uses only 100 queries per dataset, a small sample, so the coverage numbers may not hold under other query distributions. RAG-CM is an ML-plus-analytical hybrid, so its accuracy drops on hardware or operators it has not seen, and the paper gives no out-of-distribution error. The design space is also limited to the algorithm and system options they enumerate, so it does not generalize to a wholly new retrieval paradigm.