Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning
Yan Zhan, Shaobo Liu, Zhijun Gao
NeurIPS 2026
cs.AI
2026-09-28
At matched compute on Dream-7B, deterministic PRM guidance trails ORM reranking by 9.95 pp on GSM8K (12.69 pp at 32 candidates), traced to early pruning and a weak final judge.
Discrete diffusion language models (dLLMs) do not generate token by token. They hold a fully masked sequence and denoise it into an answer, so every intermediate state (the paper calls them snapshots) already exposes fragments of equations and final answers. That looks like a perfect fit for process reward model (PRM) guidance: score the evolving solution and steer compute toward promising trajectories before the answer is fixed. PRMs earned this reputation in autoregressive math reasoning, where evidence arrives in prefix order. A dLLM snapshot is a scattered subset of revealed positions, with later tokens sometimes visible while earlier ones are still masked. Work on Dream, LLaDA, and block diffusion has focused on the generator and the sampler; nobody had run the basic accounting check of whether PRM guidance beats plain independent sampling plus an outcome reward model (ORM) reranker at matched compute. This paper runs it, and the answer is no, by a wide margin.
Everything is charged in one unit: a single forward pass through a 7B-scale model. A vanilla sample costs 128 passes (the denoising length). ORM Rerank@N costs 128N denoising passes plus N scoring passes. PRM Guided branches into K candidates every 64 steps, denoises them in parallel, scores all K with the PRM, and keeps only the top one; at K=8 that is 1,040 passes against 1,032 for ORM Rerank@8, with the 0.8% surplus on the PRM side. The wall-clock version holds too: one ORM score costs about 0.3 s against 20.7 s for a full denoise. Answers are extracted with a strict regex throughout, avoiding the 11-point inflation of last-number matching.
The PRM is a frozen Dream-7B backbone with LoRA adapters and a reward head, trained on binary final-answer correctness, with each trajectory's label inherited by all of its intermediate states (cross-mask training). Controls include an ORM and a PRM trained on final states only, an SMC sampler at the same budget, and a replication on LLaDA-8B. The diagnosis separates two questions: how much pruning damages the candidate pool (measured by the Oracle ceiling) and how well the scorer makes the final pick (same pool, different scorers).
Dream-v0-Instruct-7B on all 1,319 GSM8K test problems:
| Method | N/K=8 | N/K=32 |
| Vanilla (single sample) | 43.14% | – |
| Majority@N | 60.05% | 67.63% |
| PRM Guided | 65.18% | 70.02% |
| ORM Rerank | 75.13% | 82.71% |
| Oracle ceiling | 81.05% | 91.13% |
The gap is 9.95 pp at 8 candidates and 12.69 pp at 32, and ORM Rerank@8 beats every PRM Guided budget tested, including K=32 at four times its compute. On MATH the lead is 9.85 pp (30.65% vs 20.80%), on MBPP 12.16 pp (63.04% vs 50.88%). Under wall-clock matching, ORM Rerank@6 reaches 72.40% in about 126 s per problem against 65.18% in 156 s for PRM Guided K=8.
Each failure has its own evidence. First, the signal decays with mask ratio: PRM ROC-AUC falls from 0.77 on nearly decoded states to 0.54 on mostly masked ones, and a mutual-information bound in the paper shows any scorer inherits this decay; retraining with labels from eight fresh rollouts lifts AUC by just 0.011. A top-1 cut at the earliest scored state wipes out every lineage that would later reach a correct answer in 46% of cases, dropping the pool's Oracle ceiling from 81.05% to 67.30% and unique answers per problem from 4.31 to 1.75. Second, the PRM is a poor final judge: an SMC sampler at the same budget restores the ceiling to 77.89%, yet PRM selection stalls at 65.48%; the same pool reranked by a final-state PRM hits 75.40%, level with the ORM. MBPP separates the two cleanly: its PRM reranks finished programs at 65.47%, on par with the ORM, but drops to 50.88% when guiding denoising, so the entire loss is the cost of guidance itself. On readout, a mean-pooled causal PRM manages only 0.61 ROC-AUC on final states and last-token pooling lifts it to 0.73; on 48% of test problems a wrong trajectory is confidently ranked above a correct one. LLaDA replicates the architecture effect, with bidirectional PRM guidance at 31.64% against 22.25% for causal.
For anyone doing test-time scaling on dLLMs, this is the control most guidance papers skip: charge the scorer in the same currency as the generator, and several claimed gains need re-examination. The practical recipe is concrete. Guide late and keep more than one candidate (top-4 retention at the final state cuts the kill-all-correct risk from 20% to 6%; branching every 64 steps beats every 16 by about 7 pp). Hand the final choice to a verifier trained on final states. The pool-damage versus selection-error decomposition applies to any search that prunes on intermediate scores, PRM beam search for autoregressive models included. The snapshot corpus and evaluation toolkit are public.
The authors list eight threats to validity. The load-bearing ones: evidence covers math and code only, with creative generation untested; the central result is tied to the deterministic top-1 recipe, while energy-tilted sampling, adaptive branching, and stochastic beam search remain untested, so the verdict on PRM guidance as a concept has a boundary; the ORM advantage requires a task-trained verifier, and a GSM8K-trained ORM falls to about 7% on MATH500, below Majority@32 at 17.20%; LLaDA replicates only the bidirectional-versus-causal comparison, not the ORM matchup.