How Can LLM RL Work Despite Getting Only 1 Bit per Rollout? A New Essay Tackles the Puzzle
camhowe1729 · x · 2026-09-23
Beren Millidge published a speculative essay on the information-theoretic puzzle of LLM RL: pretraining computes a loss on every token, while policy-gradient RL distills an entire rollout — often hundreds of thousands of tokens, decoded at memory-bandwidth-limited inference cost — into a single, often binary, reward. That works out to roughly 1/rolloutlength bits per sample, and crude reward functions frequently fail rollouts over trivial formatting errors.
By these arguments RL should be doubly inefficient and merely elicit behaviors already present in the base model. Yet empirically the opposite seems true, and the essay explores why RL works anyway, responding to Toby Ord's "RL doesn't have enough bits" argument. The post shares the essay and asks Ord whether he's updated his view.
More from AGI Musings
- Rishi Bommasani co-founds nonprofit STABL with SayashK and Arvind Narayanan — RishiBommasani · 2026-11-12
- Investor: AI diffusion will be slowest in consumer software — Oura, Strava, Labcorp still have no real AI strategy — MartinGTobias · 2026-11-12
- On AI Progress: When Models Lag on Your Load-Bearing Skill, You Can't See Them Improving — dfrsrchtwts · 2026-11-12
- Slack's David Hoang: agents beat people on coordination, proposes 'boundary agents' — davidhoang · 2026-11-12
- CS student mourns coding flow states he'll never experience in the AI era — chrissyykat · 2026-11-12
- Ford's 370% Turnover Led to $5 Days — But ASI Never Quits — krishnan · 2026-11-12