BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning
Quan Xiao, Mingda Liu, Gaowen Liu, Katsuki Fujisawa, Tianyi Chen
cs.AI, cs.LG, math.OC
2026-09-29
BRIDGE jointly trains retriever and LLM as bilevel RL. 3B/7B post the best average on seven open-domain QA sets, +9.6/+3.4 multi-hop EM vs the strongest baseline.
Search-R1 and VT-Search update only LLM-generated tokens: thoughts, search calls, and the final answer. Retrieved passages are treated as environment observations and masked out of the RL loss. When a rollout fails because evidence is missing or misleading, the update hits the policy instead of the retriever. That is the information-credit gap. Multi-hop QA makes it worse, because the facts live in different documents; if retrieval never surfaces them, the policy cannot learn to reason over them.
Prior work either tunes the retriever with a frozen generator or jointly trains both, without checking order. On top of VT-Search, adapting the retriever with a RAG likelihood and then running RL (RAG-then-RL) beats the reverse order. RL-then-RAG can even fall below pure RL on general QA. Retrieval has to expose useful evidence first. The LLM can only learn to search and use it afterward.
The training problem is bilevel. The lower level, for a fixed policy θ, finds retrieval parameters η that minimize the RAG negative log-likelihood of the gold answer. The upper level then updates the LLM with GRPO under that retriever, using a binary exact-match reward. The retriever stays first in the hierarchy, but it keeps moving as the policy moves.
Standard bilevel solvers do not scale here. Second-order routes need Hessian inverses. First-order penalty methods track two lower-level states, η and a penalized ηγ, which means a second copy of retriever parameters, optimizer state, and retrieval logs. BRIDGE looks at the loss landscape instead. The RAG objective still moves a lot along θ, so ∇θ LRAG cannot be treated as zero. The RL objective is comparatively flat along η. So the method keeps only the penalized retriever and steps the policy with ∇θ JRL(θ, ηγ), dropping the two RAG correction terms in the two-state estimator. Peak memory per retriever update falls 49.7% versus two-state; wall time per update falls 55.8%.
The retriever is an E5 model with query-side LoRA (rank 16); the document encoder stays frozen. It is trained with two signals: GRPO outcome credit distributed across documents by how much each one supports the gold answer, plus the RAG likelihood itself. The LLM sees only the upper-level RL loss. Retriever LoRA is updated every P steps. Training scores k=40 candidates per search; the LLM is shown the top 3.
When every rollout in a GRPO group is wrong, advantages are zero and the outcome reward teaches the retriever nothing. RAG likelihood can still rank documents by pθ(y|h,d). At training step 100, 85.26% of audited retrieval turns had zero GRPO advantage. On those turns, raw retrieval probabilities were nearly uniform, but after mixing in answer likelihood the favored document took 88% of the credit weight on average. Larger gains at 3B than at 7B fit this picture: weaker policies succeed less often, so document-level likelihood matters more.
Open-domain training uses 169,615 NQ and HotpotQA examples and the 2018 Wikipedia dump. NQ and HotpotQA are in-domain; the other five of seven benchmarks are out-of-domain. Backbones are Qwen2.5-3B/7B with an E5 retriever.
| Method | Multi-hop EM | Overall EM |
| Search-R1 (3B) | 24.9 | 33.6 |
| VT-Search (3B) | 21.5 | 34.4 |
| BRIDGE (3B) | 34.5 | 41.5 |
| Agentic-R (7B) | 39.8 | 45.0 |
| VT-Search (7B) | 39.0 | 45.9 |
| BRIDGE (7B) | 43.2 | 48.2 |
At 3B, multi-hop EM is 9.6 above Search-R1, the strongest multi-hop baseline; overall EM is 7.1 above VT-Search. At 7B, multi-hop is 3.4 above Agentic-R and overall is 2.3 above VT-Search. PopQA is the miss: 44.7 vs VT-Search 48.1 at 3B, 48.4 vs 50.2 at 7B. The 3B model beats GPT-4o, GPT-5.6 Luna, and Qwen3-235B-A22B direct inference on six of seven sets. The large models answer with no retriever, so this is not a matched comparison. It does show that coordinating a small policy with a trained retriever can beat a much larger model that answers from parameters alone.
Ablations on Qwen2.5-3B-base with group size 8: RAG-then-RL reaches 23.8 multi-hop EM, online Decoupled-RL 22.5, Decoupled-RAG 23.0, BRIDGE 31.3. Joint online updates without the hierarchy do not beat sequential RAG-then-RL. Keeping the hierarchy and adapting both online is what opens the gap.
On 4×A100 for 250 steps, VT-Search with G=8 takes 38.4 hours and scores 21.7 multi-hop EM. Doubling to G=16 takes 82.4 hours and only reaches 23.3. BRIDGE with G=8 takes 79.4 hours and reaches 31.3. Extra compute on the retriever buys more than extra rollouts. Against Agentic-R on Qwen2.5-3B-Instruct and 8×H200, BRIDGE finishes in 22.5 hours versus 43.8 and wins every dataset.
Medical evaluation uses five MIRAGE benchmarks with Qwen2.5-7B-base, still trained only on NQ and HotpotQA, retrieving from Wikipedia, PubMed, and textbooks. Macro-averaged reasoning quality / answer accuracy: BRIDGE 72.27 / 70.72, VT-Search 69.97 / 69.13, MedRAG 69.19 / 68.61. Reasoning scores come from GPT-5.6 Luna on a 0-4 rubric, scaled by 25.
If you RL a search agent and treat the retriever as a black box, credit is systematically wrong. BRIDGE is a working fix: retriever first, both components online, query-side LoRA to keep memory in check. The gains sit in multi-hop QA and smaller backbones. Popular-entity single-hop sets barely move. For a stack already running Search-R1 or VT-Search, the change is incremental: add a retriever LoRA and a RAG likelihood.
The medical numbers are zero-shot transfer. Averages lead. MedRAG uses MedCPT, a biomedical retriever, so per-dataset wins are not same-retriever comparisons.
There is no standalone Limitations section. The convergence theorem covers an idealized update with exact gradients and direct descent on η. The paper states that LoRA and the empirical surrogates used in the code are out of scope.
Rollouts allow at most two search turns, tight for multi-hop. The document encoder is frozen. Open-domain retrieval still uses the 2018 Wikipedia dump; freshness is untested. BRIDGE's wall clock is about 2× VT-Search at the same group size. Memory savings are versus the two-state bilevel estimator, not versus plain RL.
PopQA stays below VT-Search, so retriever adaptation helps little on popular-entity questions. Medical reasoning scores are LLM-as-judge, and the judge is not shown retrieved passages, so they are only loosely tied to retrieval quality. Dropping the RAG correction terms rests on RL being flat along η in this landscape. Whether that flatness transfers to other tools or corpora is untested.