It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning
Runpeng Dai, Kaili Huang, Changsung Kang, Ciya Liao
cs.IR, cs.CL
2026-09-01
Two LLMs emit keywords for queries and items, matched via an inverted index. On internal app search, F1 reaches 0.396, 10.9% above the strongest dense baseline.
First-stage retrieval decides which items ever reach ranking and auction. A miss is irreversible. BM25 matches surface terms, dense retrievers match embeddings, and generative retrievers emit document IDs. Recent LLM work mostly rewrites queries or synthesizes training data, then hands matching to a separate retriever.
The concrete question is whether LLMs can build the retrieval representations themselves, on both the query and the item side, in a form that still plugs into an inverted index. Sponsored search already prices keywords. Replacing that stack with ANN is expensive.
CoGR trains two keyword generators. A query yields a small keyword set; an item yields another. Overlap counts as a hit, and BM25 ranks the hits. Training has two phases.
Both sides optimize the same query-to-item F1. A symmetric "item-as-query" reward underperforms in ablation. Backbones are Qwen3-4B-Instruct and Qwen3-1.7B, one copy per side, five alternating rounds (10 GRPO epochs on queries, 5 on items). Decoding emits a comma-separated list with no chain-of-thought.
Two datasets. Internal app-marketplace search: 13,500/1,500 train/eval queries, 39,600 apps, about 1,000 relevant items per query. Public WANDS product search: 430/50 queries, 42,994 products, about 200 relevant items per query. Ten baselines cover sparse, dense, and generative retrieval.
| Dataset | Method | F1 |
| Internal | ANCE-Qwen4B (strongest baseline) | 0.3575 |
| Internal | CoGR-4B | 0.3963 |
| WANDS | ANCE-Qwen4B | 0.5012 |
| WANDS | CoGR-4B | 0.6819 |
That is 10.9% and 36.1% relative F1 over the strongest baseline. Query-only CoGR-4B with a frozen item side scores 0.2617 F1 on Internal; DeepRetrieval-4B scores 0.2750. Dropping SFT, sharing one generator, or swapping the item reward for transposed F1 yields 0.3751, 0.3798, and 0.3743.
Validation F1 rises from about 0.16 after SFT to 0.40 after five RL rounds, with the first round doing most of the work. Keywords get more specific: unigrams fall from 37% to 13%, phrases of three or more words rise from 12% to 31%. Feeding existing search results into the query prompt further lifts F1 to 0.4379, mainly by resolving ambiguous, misspelled, or entity-centric queries.
Teams that already run keyword indexes and keyword bidding can keep the infrastructure. Co-evolving both sides is the actual lever: a smarter query rewriter still fails if item keywords stay fixed.
The gains are real and still incremental. Ranking remains BM25. Production retrieval is usually judged on downstream business metrics that this paper does not optimize.
The authors flag two next steps: rewards tied to irrelevant-ad rate and revenue, and a stronger ranker than BM25. The internal set labels about 1,000 relevant items per query, far denser than classic IR collections, so the 36% WANDS gap may shrink on sparse labels. WANDS evaluation has only 50 queries. The item-side counterfactual reward is engineering-heavy, and the paper does not discuss vocabulary growth or index-refresh cost.