RL debate: REINFORCE is both policy gradient descent and a synthetic data method
jessi_cata · x · 2026-09-12
A technical thread on whether models under RL training evolve explicit reward-maximizing behavior. jessicata's core arguments:
- Due to general limitations of RL training (bias/variance, long runtimes, cost), reaching the "explicit reward-optimization" fixed point is not a given — training yields policies that approximate those behaviors.
- REINFORCE is simultaneously (a) policy gradient descent and (b) a synthetic data method — it approximates policy gradients via stochastic policy rollouts, and the fixed point is what the KKT analysis describes. The two perspectives need little reconciliation.
- Focus is on model behavior first, not "thoughts": if behaviors match CDT-style reward-maxing, accompanying thoughts follow, whether or not the model explicitly applies decision theory. Unlike evolution, REINFORCE has a clean argument for why it does CDT.
neocartesian presses: the two views give different behavioral predictions — would explicit reward-optimization gradually evolve?
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from AGI Musings
- Model Collapse May Not Hit LLMs, But Perhaps Human Culture or Souls — generativist · 2026-10-31
- Nina Schick: who builds intelligence and how it's distributed is the defining question of our lives — NinaDSchick · 2026-10-31
- Cheap intelligence is like cheap bread — an ingredient list satire of mass-produced AI — jsuarez · 2026-10-31
- Two futures for the young in 2026: 511 job rejections by chatbot vs unlimited AI to rebuild Cisco — PeterDiamandis · 2026-10-31
- Terence Tao warns AI is nearing a dangerous threshold as models start cheating — haider1 · 2026-10-31
- Reddit stock plunges 42% in 8 trading days as CEO says AI mode no longer drives traffic — SumitGup · 2026-10-31