Quintin Pope: RL Pathologies Stem From Online Training; Pretraining Is a Degenerate RL Case

Anthropic researcher Quintin Pope posted a series of messages on October 13 proposing a unified theoretical view of the relationship between RL and pretraining: the common pathologies of online RL methods such as RLVR (entropy collapse, reward hacking, etc.) stem mainly from "online" training itself rather than the specifics of the reward function; pretraining itself can be seen as a degenerate special case of RL — offline, applying a fixed and equal imitation strength to every trajectory. This perspective offers a new framework for understanding the failure modes of current training methods and is worth attention.

Confirmed

• Pope argues that the field habitually ties RL to online training (the model tries trajectories, gets scored, then trains on its attempts), but he contends most of this approach's pathologies come from the "online" component rather than reward function details
• He proposes that RL is essentially a form of imitation learning, with pretraining as its degenerate special case: conducted offline, with fixed and equal imitation strength per trajectory
• He believes these methods retain most of RLVR's problems even without an explicit reward function
• He envisions a hypothetical method, SIVC (Self Imitation from Verified Completions): collect trajectories and verify task completion as in RLVR, but perform pretraining-style imitation learning only on successful trajectories — reproducing RLVR's effects and problems without an explicit reward
• He also argues from a broader angle: deep learning's core lesson is that "training on data" is the most effective way to instill particular tendencies or capabilities, so RL-based alignment is hard to surpass — there may be no other way

Why it matters

• This lens places pretraining, imitation learning, and RL within a single theoretical framework; if it holds, the improvement path is to scrutinize the "online" training mechanism itself rather than endlessly tuning reward functions
• As a thought experiment, SIVC separates "verification" from "reward," helping the community pinpoint the true source of RLVR pathologies — a useful reference for future training method design

2026-10-13 ~ 2026-10-13 · 5 related posts

• [source] RL pathologies stem from online training, not reward design, researcher argues — QuintinPope5 · 2026-10-13
• Proposal: replace online RL with imitation on verified successful trajectories — QuintinPope5 · 2026-10-13
• [source] Researcher argues reward-free imitation of verified completions inherits most of RLVR's problems — QuintinPope5 · 2026-10-13
• Quintin Pope: pretraining is a degenerate special case of RL, subsumed by RL with right data — QuintinPope5 · 2026-10-13
• [source] RL alignment may be unbeatable — pretraining is just a degenerate special case of RL — QuintinPope5 · 2026-10-13