TailRL: New RL Objective Maximizes Upper-Tail Reward Coverage Instead of Just the Mean
burny_tech · x · 2026-09-07
Ruslan Salakhutdinov and collaborators released Tail-Likelihood Reinforcement Learning (TailRL), a new paper asking whether RL is optimizing the right objective.
- Motivation: Standard RL optimizes mean reward, which often collapses the policy distribution to a spike and kills probability mass on rare but high-reward rollouts — exactly the samples that matter as inference-time sampling scales up.
- Method: TailRL turns a continuous reward into a family of binary success events (exceeding a randomly chosen threshold) and maximizes the expected log-probability of exceeding it, jointly boosting mean reward and coverage of high-reward outputs. Its gradient weights rare high-reward rollouts more and can be interpreted as a mixture of Best-of-k gradients.
- Drop-in: Only a simple modification to the advantage function, compatible with existing RL pipelines.
- Results: Across object localization, maze navigation, GUI grounding, and code optimization, TailRL exploits rare high-reward samples and scales better with increased sampling.
Paper: arXiv 2609.02987, with authors including Andrea Zanette, Aarti Singh, and J. Andrew Bagnell.
Related event: CMU proposes TailRL: RL that optimizes tail rewards, not means(4 posts)→
More from Research
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Inverse-designed optical system solves MNIST using BeamZ — S_Conradi · 2026-10-27
- DeCert supervises generative reward models' rationales without reference explanations — Guiquan Sun · 2026-10-27
- VLM harnesses act as compositional generalizers, new blog post shows — a1zhang · 2026-10-27
- Article proposes a Human Genome Project for AI interpretability — ninamiolane · 2026-10-27