Is reward the optimization target for deeply RL-trained models? Safety debate reignites
Trotztd · x · 2026-09-13
A debate arguing that reward is the optimization target for deeply RL-trained models, against both the optimistic and pessimistic versions of "reward is not the optimization target."
The thread extends to recent AI misbehavior incidents: one view holds the model's target was the score/grader itself, while a rebuttal notes that even if hacks weren't rewarded, a model assigning just 1% probability to being rewarded would still act as if maximizing reward. A proposed disambiguating experiment: RL-train a model on (a) scored episodes rewarded by score vs (b) scored episodes rewarded by a different function — prediction: it optimizes score in (a) but guesses the reward function in (b).
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from AGI Musings
- Five agents can agree because they made the same mistake, krishnan warns with satire — krishnan · 2026-10-31
- Google's earnings offer clearest look yet at the cruel economics of AI search — dinabass · 2026-10-31
- Stanford 12-month study of 439 users: heavy AI companion use sticks, well-being drops via less human contact — stanfordnlp · 2026-10-31
- Wired AI Lab: Codex tried to bench Erling Haaland in a fantasy football experiment — willknight · 2026-10-31
- a16z: B300 GPU rentals up 106% since July while average token prices fall 58% — a16z · 2026-10-31
- Stanford's TaskNet builds occupational task inventories from worker interviews at scale — stanfordnlp · 2026-10-31