Meta team proposes MaD-RL, an RL framework for matching target output distributions in LLMs
nagpalchirag · x · 2026-09-24
A team from Meta has released MaD-RL (Matching Distributions for Calibrating LLMs with RL), a general post-training framework that uses reinforcement learning for distribution matching.
The problem: standard RL post-training like GRPO maximizes per-output reward, which concentrates probability on a single mode and hurts diversity — yet applications like synthetic data generation, fairness constraints, and policy exploration require controlling the mix of languages, solution strategies, styles, or generated demographics across outputs.
Key contributions:
- Shows GRPO-style recipes collapse toward a single mode; entropy regularization and temperature help only marginally and can't target arbitrary mixtures
- Observes prior work is a special case of distribution matching with L2 divergence
- Derives reward functions for KL and Jensen-Shannon divergences with theoretical justification
- Validates the approach on math reasoning and programming experiments
The paper is available on arXiv with authors including Sourabh Kulkarni and Chirag Nagpal.
Related event: Meta Team Proposes MaD-RL to Calibrate LLM Output Distributions with RL(2 posts)→
More from Research
- Q Labs: LLMs are depth-bottlenecked, loss keeps improving to 128 layers — rickasaurus · 2026-09-24
- OverclaimBench: coding agents never opened files in 68% of 1,140 review runs — hugo_larochelle · 2026-09-24
- TANGO: Whole-Body VLA Maps RGB to 29-DoF Actions for Humanoid Navigation, Trained Fully in Simulation — berkeley_ai · 2026-09-24
- Theory of Agent survey from Beihang and 7 institutions reviews ~600 agent studies — jiqizhixin · 2026-09-24
- GPT-6 Astra cracks Erdős–Sós graph conjecture, proof verified in Lean — IgorCarron · 2026-09-24
- NEJM AI: practical advice on AI's impact on scientific thinking and mentorship — zakkohane · 2026-09-24