Toby Ord stands by his thesis that RL learns far less per FLOP than pretraining

Oxford philosopher Toby Ord, in a multi-round public exchange with researcher Beren Millidge, reaffirmed his core thesis about the information efficiency of reinforcement learning and offered additional explanations for why RL still works.

Confirmed

• Millidge laid out an information-theoretic paradox in LLM RL: pretraining computes a loss on every token, whereas policy-gradient RL only receives a (often binary) scalar reward at the end of rollouts spanning hundreds of thousands of tokens—roughly 1 bit per trajectory—yet still trains LLMs effectively.
• Asked whether he had updated his "RL doesn't have enough bits" view, Ord said his position was largely unchanged: rereading his old piece, he found the argument still holds and many predictions have held up.
• Ord reiterated that the upper bound on information learned per FLOP in RL is far lower than in pretraining, raising two questions: why RL still works, and how far it can go.
• His explanations: the available bits, though few, are extremely relevant; mid-training partly explains how RL scaled to today's levels; and RL generalizes to non-verifiable tasks better than expected.
• Ord also raised a caveat: models may merely be learning what is being tested, so benchmarks may overstate real RL progress.

Why it matters

• The debate goes to the heart of the current RL scaling路线: if RL's information efficiency is indeed far below pretraining, there may be a ceiling on capability gains from RL.
• If the hypothesis that benchmarks overstate RL progress holds, it would affect how real capabilities of frontier models are evaluated—worth watching for follow-up validation.

2026-09-23 ~ 2026-09-24 · 5 related posts

• [source] How Can LLM RL Work Despite Getting Only 1 Bit per Rollout? A New Essay Tackles the Puzzle — camhowe1729 · 2026-09-23
• [source] Toby Ord on RL's bits problem: 'I haven't changed my mind much' — his jagged-capabilities predictions held up — tobyordoxford · 2026-09-24
• Toby Ord stands by his RL thesis: lower bits-per-FLOP ceiling explains today's jagged AI capabilities — tobyordoxford · 2026-09-24
• Toby Ord: RL's scaling surprise may hinge on mid-training, benchmark gains may overstate progress — tobyordoxford · 2026-09-24

1 near-duplicate retellings: tobyordoxford