Meta team proposes MaD-RL, an RL framework for matching target output distributions in LLMs
nagpalchirag · x · 2026-09-24
A team from Meta has released MaD-RL (Matching Distributions for Calibrating LLMs with RL), a general post-training framework that uses reinforcement learning for distribution matching.
The problem: standard RL post-training like GRPO maximizes per-output reward, which concentrates probability on a single mode and hurts diversity — yet applications like synthetic data generation, fairness constraints, and policy exploration require controlling the mix of languages, solution strategies, styles, or generated demographics across outputs.
Key contributions:
- Shows GRPO-style recipes collapse toward a single mode; entropy regularization and temperature help only marginally and can't target arbitrary mixtures
- Observes prior work is a special case of distribution matching with L2 divergence
- Derives reward functions for KL and Jensen-Shannon divergences with theoretical justification
- Validates the approach on math reasoning and programming experiments
The paper is available on arXiv with authors including Sourabh Kulkarni and Chirag Nagpal.
Related event: Meta Team Proposes MaD-RL to Calibrate LLM Output Distributions with RL(2 posts)→
More from Research
- AlphaGenome2 hits SOTA linking distal enhancers to target genes from gradients — anshulkundaje · 2026-11-12
- AlphaGenome2 ablations: early encoder conditioning nearly doubles eQTL recall — anshulkundaje · 2026-11-12
- Author says his CHI2027 ADR paper was better than papers he reviewed — windx0303 · 2026-11-12
- Google launches AlphaGenome 2 to predict how DNA variants affect specific cell types — sundarpichai · 2026-11-12
- Stanford's James Zou joins The Economist podcast on AI agents in science — anshulkundaje · 2026-11-12
- New learning rule boosts Hopfield network memory capacity over 10×, per Nature Communications study — bravo_abad · 2026-11-12