RL debate: REINFORCE is both policy gradient descent and a synthetic data method
jessi_cata · x · 2026-09-12
A technical thread on whether models under RL training evolve explicit reward-maximizing behavior. jessicata's core arguments:
- Due to general limitations of RL training (bias/variance, long runtimes, cost), reaching the "explicit reward-optimization" fixed point is not a given — training yields policies that approximate those behaviors.
- REINFORCE is simultaneously (a) policy gradient descent and (b) a synthetic data method — it approximates policy gradients via stochastic policy rollouts, and the fixed point is what the KKT analysis describes. The two perspectives need little reconciliation.
- Focus is on model behavior first, not "thoughts": if behaviors match CDT-style reward-maxing, accompanying thoughts follow, whether or not the model explicitly applies decision theory. Unlike evolution, REINFORCE has a clean argument for why it does CDT.
neocartesian presses: the two views give different behavioral predictions — would explicit reward-optimization gradually evolve?
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from AGI Musings
- Security researcher: vibe coding will force software to consolidate around mature products — jd_pressman · 2026-10-31
- AI pushes circuit lower bound from 3.1n to 3.5n as UCLA-backed challenge aims at P vs NP — soumitrashukla9 · 2026-10-31
- What is work for? Cross-national survey of 8,234 adults on meaning in the AI age — soumitrashukla9 · 2026-10-31
- Which universities win top students may hinge on healthy AI-use boundaries, argues researcher — brianryhuang · 2026-10-31
- Researchers debate whether more inference compute can substitute for training compute — teortaxesTex · 2026-10-31
- Five agents can agree because they made the same mistake, krishnan warns with satire — krishnan · 2026-10-31