Tail-Likelihood RL paper targets upper-tail rewards that mean-based RL masks
burkov · x · 2026-09-07
RL for generative models typically maximizes expected reward, but policies with similar means can differ sharply in the probability of producing rare, high-value outputs — a gap that becomes decisive when sampling scales up during training or inference.
The paper introduces Tail-Likelihood Reinforcement Learning for continuous rewards: rather than optimizing a single mean, it treats exceeding each reward threshold as a binary success and maximizes the expected log-probability of success across thresholds drawn uniformly from the reward range. The resulting objective decomposes into a harmonic mixture of Best-of-k gradients, preserving upper-tail coverage while improving performance.
Related event: CMU proposes TailRL: RL that optimizes tail rewards, not means(4 posts)→
More from Research
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27
- Marin's dlwh Explains the Scaling Law Fit: Estimating the Loss Floor Early — dlwh · 2026-10-27
- Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2% — dlwh · 2026-10-27
- Marin Shares Forecast-vs-Measured Plots: Paloma Hits 0.4% RMS Error — dlwh · 2026-10-27