Toy Example Shows Why Synthetic-Data RL Converges to Maximization Behavior
jessi_cata · x · 2026-09-12
jessicata offers a simple toy example arguing that synthetic-data-style RL inevitably yields maximization behavior at the fixed point:
- Sample 1000 tokens from the model's next-token distribution;
- Score each with f(token) ∈ [0,1] and keep samples with probability f(token);
- Fine-tune on the remainder and repeat;
- Iterating drives the model to put all its mass on top-scoring tokens.
A concise rebuttal in the "Reward is not the optimization target" debate, highlighting the gap between sample-filter-tune loops and true RL.
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from Research
- AISTATS 2027 workshop proposals due Nov 6 after one-week extension — qberthet · 2026-10-31
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- PI agent rejected 75 proposals in 30 hours; manager banned recurring bad ideas — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31
- natolambert: importance-sampling fixes for train-inference mismatch dominate RL research — natolambert · 2026-10-31