Why RL Works for LLMs Despite Offering Only 1 Bit of Signal per Rollout
burny_tech · x · 2026-08-18
RL training for LLMs is famously informationally inefficient: each rollout yields a single scalar (often binary) reward — roughly 1/rolloutlength bits per sample — and rollouts cost more compute than prefill, while reward functions are often crude.
Yet a blog by Beren Millidge offers an intuitive answer: SNR. Unlike SFT's dense loss on every token, RL's sparse bits carry high signal-to-noise directional information, so tens to hundreds of steps often dramatically improve performance — far fewer than SFT needs.
Related event: Why Reinforcement Learning Works for LLMs: Insights from New Analyses(3 posts)→
More from Research
- Researcher slams papers whose stacked changes beat even weak baselines barely — retr0jirachi · 2026-10-16
- FrankenSim builds a physics-based virtual orchestra, piano and drum demos live — doodlestein · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- ApolloPFN: a zero-shot forecasting network that natively handles exogenous covariates — burkov · 2026-10-16
- ICLR reviewer says 4/5 papers in review pile read like AI slop — berkustun · 2026-10-16
- Tsinghua's DBS learns content-adaptive frequency band splits for audio source separation — thu-ml · 2026-10-16