Why RL Works for LLMs Despite Offering Only 1 Bit of Signal per Rollout

burny_tech · x · 2026-08-18

RL training for LLMs is famously informationally inefficient: each rollout yields a single scalar (often binary) reward — roughly 1/rolloutlength bits per sample — and rollouts cost more compute than prefill, while reward functions are often crude.

Yet a blog by Beren Millidge offers an intuitive answer: SNR. Unlike SFT's dense loss on every token, RL's sparse bits carry high signal-to-noise directional information, so tens to hundreds of steps often dramatically improve performance — far fewer than SFT needs.

Original post →

Why RL Works for LLMs Despite Offering Only 1 Bit of Signal per Rollout

More from Research