RL Researchers Debate: REINFORCE as Both Policy Gradient and Synthetic Data Method

On September 12, 2024, researchers engaged in a technical debate centered on the question of whether models trained with RL evolve explicit reward-maximizing behavior. The main participants were jessicata and neocartesian, and the discussion focused on two perspectives for understanding the REINFORCE algorithm and its convergence properties.

Confirmed

• jessicata's core argument: due to inherent limitations of RL training (bias/variance issues, long training times, high costs), training does not necessarily converge to a fixed point of explicit reward optimization, but REINFORCE shows how to approximate policy gradients using random rollouts of the policy in the environment.
• To clarify fixed-point behavior, jessicata offered a simple toy example: sample the model's next-token distribution 1000 times, score each sample with a scoring function f(token)∈[0,1], keep each sample with probability f(token), then fine-tune the model to favor the retained samples. Based on this, jessicata argued that "synthetic-data-style RL produces maximizing behavior at the fixed point."
• When asked how to reconcile the decision-theoretic view of RL with their own mental model of "goal-directed synthetic data generation," jessicata stated clearly that the two views are not contradictory: REINFORCE's random rollouts can of course be seen as synthetic data, and the policy-gradient and synthetic-data perspectives can be unified within fixed-point analysis.
• neocartesian responded to the essay "Reward is not the optimization target" with a decision-theoretic analysis of RL, showing that idealized policy-gradient optimization converges stably to KKT points of the policy-to-expected-reward mapping, and connected the problem to equilibria of imperfect-recall games.

Why it matters

This debate touches on a fundamental question of RLHF/RL fine-tuning: what does a model actually optimize after training—explicit reward maximization, or merely imitation of filtered data? jessicata's toy example provides an intuitive argument that synthetic-data-style RL inevitably leads to maximizing behavior, while neocartesian's analysis via KKT points and game equilibria offers a more formal framework. The convergence of these two approaches helps clarify the theoretical foundations of safety issues such as reward hacking, and offers insights into the long-term behavior of RL training for large models.

2026-09-12 ~ 2026-09-12 · 5 related posts

• [source] RL Theory Debate: Policy Gradients Converge at KKT Points, Not Reward Targets — neocartesian · 2026-09-12
• [source] REINFORCE's two views don't conflict: policy gradients and synthetic data unify at the fixed point — jessi_cata · 2026-09-12
• [source] RL debate: REINFORCE is both policy gradient descent and a synthetic data method — jessi_cata · 2026-09-12
• A toy example of why RL fine-tuning concentrates mass on max-reward tokens — jessi_cata · 2026-09-12
• Toy Example Shows Why Synthetic-Data RL Converges to Maximization Behavior — jessi_cata · 2026-09-12