MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment
Tzu-I Ho, Yung-Yu Shih, Shang-Yu Su, Dongzhe Wang, Yun-Nung Chen
cs.IR, cs.CL
2026-09-29
Three 7-8B LLMs expand a query and a 14B model merges them. BM25 nDCG@10 rises 2.1-14.9 on five BEIR sets; prompts are auto-tuned on retrieval scores.
Query2Doc and HyDE ask an LLM to turn a short user query into a pseudo-document or a hypothetical answer so BM25 can match the corpus vocabulary. One model tends to expand in a narrow house style: it drops paraphrases, invents details, or locks an ambiguous query onto a single reading. The prompts that steer this behavior also fail to travel. A template that works on Qwen2.5 usually has to be rewritten for Llama-3.1 or Mistral. Add another generator and the prompt work grows with the pool, which is why multi-LLM ensembles rarely look cheap for retrieval.
That failure mode is real. Swap the generator in a live search stack and both the expansion style and the usable prompt often break together.
MERGE hits both issues. Three heterogeneous 7-8B open-source models write independent expansions; a 14B model fuses them into one query. Every prompt, generators and fusion alike, is tuned automatically against BM25 nDCG@10.
Inference is two stages.
The fused text goes to a frozen BM25 index. No re-indexing, no rank fusion, one lexical pass. Following common BM25 practice, the original query is repeated several times and the enrichment is appended. The output is plain text, so a dense retriever could consume it; every reported number uses BM25.
Prompts are searched offline with task-grounded APO. The loop starts from SPO, which has an optimizer LLM rewrite a prompt and an evaluator LLM pick a winner. MERGE replaces the evaluator with the actual IR metric: nDCG@10 of BM25 on a fixed 20% labeled training subset. Each round is a small tournament. Round 0 samples two expansions from the seed; the higher score becomes champion. Later rounds have the optimizer rewrite the champion into a new draft, sample two expansions from that draft, and keep the best of the two challengers and the incumbent. A challenger replaces the whole champion tuple (expansion, score, prompt). The loop stops when the champion survives two consecutive rounds. The optimizer sees the score gap, a 95% CI, a p-value, and examples where a challenger helped or hurt retrieval. MERGE (APO) uses only the current-round signal. MERGE (APO w/ history) also feeds the last five rounds. APO tunes the three generators first, then freezes them and tunes Stage 2. Most tournaments finish in under 10 rounds, typically under an hour per (stage, model) pair on one H100, paid once and reused.
Five BEIR tasks cover open-domain QA, scientific claims, financial opinions, argument retrieval, and entity retrieval. The metric is nDCG@10. Every row shares BM25 and differs only in the query text. docT5query is a supervised, corpus-side baseline. Exp4Fuse runs two BM25 passes (raw query and an LLM pseudo-document) and fuses the lists. Those baseline numbers are copied from Liu and Zhang (2025); MERGE did not rerun them.
| Method | NQ | SciFact | FiQA | Touché-2020 | DBPedia |
| Vanilla BM25 | 30.48 | 67.62 | 23.85 | 44.23 | 31.99 |
| docT5query | 38.10 | 67.50 | 25.20 | 34.70 | 33.10 |
| BM25+Exp4Fuse | 39.10 | 68.80 | 24.70 | 51.20 | 36.10 |
| docT5query+Exp4Fuse | 42.80 | 71.30 | 26.30 | 39.90 | 38.90 |
| MERGE w/o APO | 34.52 | 67.79 | 19.41 | 38.69 | 33.59 |
| MERGE (APO) | 45.42 | 71.40 | 25.59 | 50.16 | 37.94 |
| MERGE (APO w/ history) | 42.50 | 70.01 | 25.91 | 51.67 | 37.84 |
Against the raw query, all five datasets move up: +14.9 on NQ (30.48 to 45.42), +7.4 on Touché-2020, +6.0 on DBPedia, +3.8 on SciFact, +2.1 on FiQA (23.85 to 25.91). Against BM25+Exp4Fuse, MERGE wins all five, from +0.5 on Touché-2020 to +6.3 on NQ. Against the stronger stack docT5query+Exp4Fuse it wins on NQ (+2.6) and Touché-2020 (+11.8), ties SciFact (+0.1), and loses on FiQA (-0.4) and DBPedia (-1.0).
Unsupervised expansion baselines lose on 5/5. The supervised-plus-fusion stack is matched or beaten on 3/5.
Seed prompts can hurt: FiQA drops 4.4 points versus vanilla BM25, Touché-2020 drops 5.5. APO turns both into gains. Versus MERGE without APO, the better variant adds 10.9 (NQ), 3.6 (SciFact), 6.5 (FiQA), 13.0 (Touché-2020), and 4.4 (DBPedia). Entity-heavy sets (NQ, SciFact, DBPedia) prefer APO without history; broader FiQA and Touché-2020 prefer the history variant.
Stage 2 is doing real work. On NQ the three Stage-1 expansions score 38.79, 34.59, and 41.06; the fused query scores 45.42, +4.4 over the best generator. SciFact adds 0.9, FiQA adds 0.6. For the NQ query "the cuban revolt against spain was led by", Llama names José Martí and 1895-1898, Mistral supplies the 1868-1898 conflict window, and Qwen mostly emits generic verbs. The 14B fusion keeps Martí, Gómez, and Maceo, the date span, and the split between ideological and military leadership.
This is an engineering increment on query-side expansion, not a new retrieval architecture.
If the stack is already BM25, prompt retuning becomes a one-off tournament rather than a per-model rewrite, and serving costs three 7-8B calls plus one 14B call and a single BM25 pass. There is no T5 pass over the corpus and no re-index or fusion weights. When the corpus changes, docT5query has to relabel every document; MERGE only touches the query.
Four LLM calls are not free, and APO still wants a labeled 20% subset. The fit is lexical search where query wording sits far from the documents and a dense retriever or reranker is not on the table yet. The conclusion mentions that the fused query could feed a dense retriever. That number is not in the tables.
There is no standalone Limitations section before the conclusion. The gaps sit in the experimental design.
All headline numbers are BM25. Dense retrieval, reranking, and passage-side enrichment appear only as future remarks in the conclusion. APO scores prompts on a labeled 20% subset, so calling the pipeline zero-shot unsupervised overstates what the loop needs. Exp4Fuse and docT5query scores are imported; decoding and model-family match cannot be checked from the text. Latency and token cost have no comparison table, so four generations per query may or may not be acceptable in production. Qwen2.5-14B is both the fusion model and the APO optimizer; there is no ablation for self-preference. Some converged prompts read as bags of model-specific steering tokens, which the text itself flags. The expansions stay readable; the prompts are hard to audit by hand. Neither APO variant wins everywhere, and the paper does not give a routing rule. The Stage-2 ablation in Table 2 covers only NQ, SciFact, and FiQA.