RLVR fine-tuning collapses solution space, replay buffer fix boosts diversity and pass@8
manoelribeiro · x · 2026-10-15
A research thread reveals a systematic flaw in today's RLVR fine-tuning and proposes a simple fix.
- Problem: RLVR causes solution-space collapse—early answers get reinforced in a winner-takes-all dynamic, wiping out other valid approaches
- Evidence: The new ModeBench benchmark shows frontier models solve toy problems one way but never recover all solutions even at 512 samples per prompt; 56%-98% of verified modes in some domains are never produced. Small Qwen-Instruct models show richer solution diversity than frontier models
- Fix: Add a replay buffer per accepted solution mode (Re:Max, Re:Dr) to preserve existing solutions while exploring new ones
- Results: Higher solution diversity and pass@8 accuracy across model families, sizes, and RL objectives; replay also keeps inference robust when a solution option is withdrawn
Related event: RLVR Fine-Tuning Collapses Solution Diversity; Replay Buffers Fix It(2 posts)→
More from Research
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- 8 non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15