A control experiment to test whether RL models converge to reward-seeking
jessi_cata · x · 2026-09-14
Proposing a disambiguating experiment on whether reward is the optimization target for deeply RL-trained models: train on (a) scored episodes rewarded by score, (b) scored episodes rewarded by some other function, then deep-fry the model with RL. Prediction: it converges to reward-seeking — optimizing score in (a) but guessing the reward function in (b).
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from AGI Musings
- Five agents can agree because they made the same mistake, krishnan warns with satire — krishnan · 2026-10-31
- Google's earnings offer clearest look yet at the cruel economics of AI search — dinabass · 2026-10-31
- Stanford 12-month study of 439 users: heavy AI companion use sticks, well-being drops via less human contact — stanfordnlp · 2026-10-31
- Wired AI Lab: Codex tried to bench Erling Haaland in a fantasy football experiment — willknight · 2026-10-31
- a16z: B300 GPU rentals up 106% since July while average token prices fall 58% — a16z · 2026-10-31
- Stanford's TaskNet builds occupational task inventories from worker interviews at scale — stanfordnlp · 2026-10-31