Toy Example Shows Why Synthetic-Data RL Converges to Maximization Behavior
jessi_cata · x · 2026-09-12
jessicata offers a simple toy example arguing that synthetic-data-style RL inevitably yields maximization behavior at the fixed point:
- Sample 1000 tokens from the model's next-token distribution;
- Score each with f(token) ∈ [0,1] and keep samples with probability f(token);
- Fine-tune on the remainder and repeat;
- Iterating drives the model to put all its mass on top-scoring tokens.
A concise rebuttal in the "Reward is not the optimization target" debate, highlighting the gap between sample-filter-tune loops and true RL.
More from Research
- kalomaze: You don't need an analytic transfer theory, just a learnable transfer-extrapolation function — kalomaze · 2026-09-12
- kalomaze: Information asymmetry, not verifiability, is the general primitive behind RLVR gains — kalomaze · 2026-09-12
- Simons Institute holds workshop on AI's rapid acceleration of mathematics and theoretical CS — jasondeanlee · 2026-09-12
- Dev Fine-Tuned a 2B LLM on WhatsApp Group Chat, Simulating Six Friends on an M1 Pro — BarisSayit · 2026-09-12
- First quantitative evidence: Claude and GPT now use GUIs as well as APIs — ysu_nlp · 2026-09-12
- World Models Will Power the Next Leap in AI Agents — And They May Never Show Video — furongh · 2026-09-12