A toy example of why RL fine-tuning concentrates mass on max-reward tokens
jessi_cata · x · 2026-09-12
jessicata offers a simple toy example of RL fine-tuning's fixed point: sample a model's next-token distribution 1000 times, score tokens by some f(token) in [0,1], keep samples with probability f(token), then fine-tune the model toward the remainder. Iterating this drives nearly all probability mass onto tokens with maximum f — the reward-maxing fixed point.
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from Research
- AISTATS 2027 workshop proposals due Nov 6 after one-week extension — qberthet · 2026-10-31
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- PI agent rejected 75 proposals in 30 hours; manager banned recurring bad ideas — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31
- natolambert: importance-sampling fixes for train-inference mismatch dominate RL research — natolambert · 2026-10-31