Researchers Debate Whether Reward Is the Optimization Target in Deep RL

Starting September 12, 2024, researchers engaged in a multi-day technical debate over whether RL-trained models develop explicit reward-maximizing behavior and whether reward is the optimization target of deep RL models. Key participants included jessicata, neocartesian, and sofgreyu47, with discussion extending to two views of REINFORCE, convergence properties, and an actionable discriminating experiment.

Confirmed

• jessicata argued that due to RL's inherent limitations (bias/variance, long training times, high cost), training does not necessarily converge to a fixed point of explicit reward optimization, though REINFORCE shows how to approximate policy gradients via random rollouts.
• jessicata offered a toy example: sample the model's next-token distribution 1000 times, score tokens with f(token)∈[0,1], keep each sample with probability f(token), then fine-tune the model toward retained samples—arguing synthetic-data-style RL produces maximization behavior at the fixed point.
• Asked how to reconcile the decision-theoretic view of RL with a goal-directed synthetic-data mental model, jessicata said the two are consistent: REINFORCE's random rollouts can be viewed as synthetic data, unified under fixed-point analysis.
• neocartesian responded to "Reward is not the optimization target" with a decision-theoretic analysis: ideal policy-gradient optimization converges stably at KKT points of the policy-to-expected-reward mapping, linking the problem to equilibria of imperfect-recall games.
• On September 13, jessicata countered the alignment-community debate, arguing reward is itself the optimization target for deeply RL-trained models—opposing both optimistic and pessimistic variants of the "reward is not the target" position. Trotztd relayed the heated debate.
• sofgreyu47 disagreed, arguing that calling reward the optimization target conflates the training objective with what the model actually learns to optimize—a distinction that grows more important as models become more capable.
• On September 14, jessicata proposed a discriminating experiment: design two types of scored episodes—(a) reward directly determined by the score, (b) score present but reward from another function—then apply intensive RL training, predicting the model converges toward reward rather than score.

Why it matters

The debate touches the foundational question of RLHF/RL fine-tuning: after training, do models explicitly maximize reward or merely imitate filtered data? jessicata's toy example and controlled experiment provide an intuitive argument and testable path; neocartesian's KKT and game-equilibrium analysis offers a more formal framework; and sofgreyu47's distinction between training objectives and learned objectives signals potential divergence in the era of strong models. These converging approaches help clarify the theoretical basis of reward hacking and long-run behavior of large-model RL training.

2026-09-12 ~ 2026-09-14 · 9 related posts

• RL Theory Debate: Policy Gradients Converge at KKT Points, Not Reward Targets — neocartesian · 2026-09-12
• REINFORCE's two views don't conflict: policy gradients and synthetic data unify at the fixed point — jessi_cata · 2026-09-12
• [source] RL debate: REINFORCE is both policy gradient descent and a synthetic data method — jessi_cata · 2026-09-12
• A toy example of why RL fine-tuning concentrates mass on max-reward tokens — jessi_cata · 2026-09-12
• Toy Example Shows Why Synthetic-Data RL Converges to Maximization Behavior — jessi_cata · 2026-09-12
• [source] Researcher argues reward is the optimization target for deeply RL-trained models — jessi_cata · 2026-09-13
• Is reward the optimization target for deeply RL-trained models? Safety debate reignites — Trotztd · 2026-09-13
• RLHF debate: conflating reward with what models actually optimize gets riskier as they scale — sofgrey_u47 · 2026-09-13
• [source] A control experiment to test whether RL models converge to reward-seeking — jessi_cata · 2026-09-14