PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
MinKeon Kim, Namjun Lee, Jaekwang Kim
EMNLP 2026
cs.CL, cs.AI
2026-08-31
A PRM scores RAG steps for logic and evidence, then DPO trains on tree-search pairs. Qwen2.5-7B hits 34.5 EM / 44.1 F1 on five QA sets, above Search-R1, using 5k questions.
Multi-hop RAG ties retrieval to reasoning. One bad search early on poisons the rest of the chain. Outcome-only methods such as Search-R1 reward the final answer and never punish a wasted retrieval. The signal is also sparse, so they tend to need a lot of data.
Existing process rewards still do not score the step. They assign credit from final F1, so a trajectory that retrieved the wrong passage and lucked into the right answer looks positive. A RAG step has to be logically coherent and grounded in the documents just fetched. Math PRMs only check the first of those.
PRO-Step trains a generative process reward model, then uses it to build step-level preference pairs via value tree search, then trains the policy with DPO.
Trajectories interleave reasoning in think blocks with search calls. The environment appends a documents block and generation continues until an answer. From 2,000 HotpotQA and MuSiQue questions the pipeline samples 16 trajectories each, keeps 31,728 complete ones, and has QwQ-32B write a rationale plus a 0/1 label per step. Later steps can recover by rewriting the query, so a local miss does not discard the suffix.
The PRM starts from DeepSeek-R1-Distill-8B and emits rationale then label. In the search tree, non-terminal value is the PRM bit times a depth discount; terminals use discounted F1. Sibling nodes are ranked by backup quality plus PRM score; pairs with a large gap become chosen/rejected. DPO is applied at the diverged step only, with documents blocks masked so the policy does not memorize retrieved text.
The main policy is Qwen2.5-7B-Instruct. Retrieval is the 2018 Wikipedia dump (5.9M docs) with BGE-base-en-v1.5 and k=3. Policy training uses 5,000 multi-hop questions: 2,000 each from HotpotQA and MuSiQue, 1,000 from 2WikiMultiHopQA.
PRO-Step posts the best average EM and F1 among the methods in the table.
| Method | Mean EM | Mean F1 |
| Standard RAG | 23.8 | 31.6 |
| Search-R1 | 32.0 | 40.7 |
| ReasonRAG | 32.6 | 41.0 |
| StepSearch | 33.2 | 42.0 |
| PRO-Step | 34.5 | 44.1 |
Versus Search-R1 the gap is +2.5 EM and +3.4 F1, widening to +9.2 EM and +8.9 F1 on 2WikiMultiHopQA. Search-R1 reports about 170k training instances; PRO-Step uses 5k seed questions. Dropping the PRM and pairing from depth-discounted F1 alone falls to 32.7 EM / 41.9 F1. On the same pairs, DPO beats SFT and KTO; KTO averages only 28.0 EM.
With 128 candidates, weighted majority voting using this RAG PRM reaches 46.5 / 59.5 / 50.8 F1 on 2Wiki / HotpotQA / PopQA, above majority vote, VersaPRM, and MathPRM. The general PRMs pick fluent but ungrounded traces as the pool grows. Llama-3.1-8B-Instruct moves from 5.08/13.08 to 16.16/22.57; at 3B, PRO-Step beats Search-R1 31.04/40.00 versus 26.22/33.51.
Process supervision for agentic RAG needs an evidence check, not only a logic check, and that signal can be baked into the policy so inference does not call extra judges. Data efficiency is the practical hook: 5k questions beating an outcome RL run that used about 170k. Code, models, and data are public.
The gain is real at 7B on Wikipedia retrieval. It is not the kind of jump you get by swapping the retriever or the backbone.
The labeler is QwQ-32B, not GPT-4o. A 500-sample audit against Claude Opus 4.7 gives κ=0.6104, substantial agreement with leftover noise. On MuSiQue, PRO-Step trails StepSearch by 1.3 EM; the paper blames the open-source teacher and 15,877 pairs versus StepSearch's GPT-4o-distilled 19k corpus. Reproducibility is bought with some headroom.
Bamboogle has 125 test items. The paired t-test versus ReasonRAG is p=0.66, so that gap is noise. PRM-feedback regeneration dropped 10.7 EM on average because regenerated traces drifted from the inference-time search format. All main numbers use 2018 Wikipedia and k=3; live web search is untested.