ModeBench shows RLVR collapses LLMs into single-solution mode; Re:Max training preserves diversity
manoelribeiro · x · 2026-10-15
A team led by Liv Daliberti (with Marwa Abdulhai, Peter Henderson, Sofiia Druchyna) released ModeBench and the Re:Max training method to study solution diversity in LLMs:
- Key finding: Echoing George Pólya's "better to solve one problem five ways than five problems one way," RLVR pushes LLMs in exactly the opposite direction — toward a single solution per problem. On graph coloring, Dr.GRPO training causes Qwen2.5-3B's 32 samples to collapse onto one dominant solution, while Re:Max improves both accuracy and diversity.
- ModeBench: A benchmark of multi-solution problems across 5 domains (graph coloring, Countdown, executable Python, algebra, constraint optimization), with 72 frozen splits and 15,552 rows; a verifier checks both correctness and solution-mode uniqueness, and evaluation runs on CPU without PyTorch or CUDA.
- Re:Max / Re:Dr: A replay-based approach keeping models robust when the solution space shifts, e.g., when a preferred solution becomes invalid.
Dataset and Re:Max implementation are open-sourced (Apache-2.0). The project lead is looking for internships.
Related event: RLVR Fine-Tuning Collapses Solution Diversity; Replay Buffers Fix It(2 posts)→
More from Models
- Google says teams across the company are using Gemini 4 Argon on its hardest engineering problems — CurieuxExplorer · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Leak: Moonshot's Kimi flagship 'x4' done with pre-training, K3.1 in internal testing — lyraxana · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- NVIDIA ships two open Nemotron 3.5 Bolt models: a 74B-A9B MoE and a mobile-friendly E4B — HankYeomans · 2026-10-15