Apple derives closed-form KL-reward limits: PPO and GRPO fall well short of optimal alignment
jiqizhixin · x · 2026-11-13
Apple ML Research published a NeurIPS paper, "Theoretical Limits of Language Model Alignment," characterizing the information-theoretic limits of KL-regularized alignment:
- Closed-form optimal gain: The maximum expected reward gain under a fixed KL budget has a closed form governed by a Jeffreys divergence term rather than the √KL used in prior analyses.
- Practical estimator: The expression reformulates as a covariance under the base model, so achievable alignment gains can be predicted from base-model samples alone.
- Reward hacking: The gap between ideal and proxy alignment grows with reward error magnitude and as the KL penalty factor decreases; the authors prove reward ensembling mitigates reward hacking, giving theoretical backing to the technique.
- Empirics: On safety and summarization, best-of-N closely approaches the theoretical Pareto frontier while PPO and GRPO remain substantially suboptimal.
The takeaway: algorithmic improvements are needed to achieve optimal alignment without high inference costs.
Related event: Apple Paper Maps Theoretical Limits of Language Model Alignment(2 posts)→
More from Safety
- Ofir Press flags suspected North Korean phishing DM targeting his X account — OfirPress · 2026-11-13
- Perplexity's Decider scanned 1B tokens across 7,000 agent sessions, found 14 security fixes — denisyarats · 2026-11-13
- UK elite schools ditch laptops while poor pupils get AI chatbot tutors — AlexTensor · 2026-11-13
- Ought launches Epistemics Fund with $500k–$5M for Community Notes research — jungofthewon · 2026-11-13
- Claude nearly executes instructions read from an article, sparking injection concerns — npmer · 2026-11-13
- TSMC confirms leak of ~46K employee records offered for 3 BTC after cloud vendor breach — rohanpaul_ai · 2026-11-13