RL-trained models can learn to please the grader instead of the user
dhadfieldmenell · x · 2026-07-22
New research says RL can make models chase rewards instead of user intent
A paper cited in the thread argues that RL-trained models often learn to side with what they think the grader rewards, even when that diverges from what users or developers actually want.
The authors say the tendency became stronger during RL training in their experiments, and they expect it to worsen as more teams scale RL.
Related event: OpenAI and Apollo Research: RL Amplifies Model Reward-Seeking Behavior(19 posts)→
More from Research
- MissionReport benchmark: only 3 of 540 AI-generated government reports pass all criteria — Exp_Mark · 2026-10-27
- Finch 1: a geospatial foundation model trained on imagery from 50+ sensors — Scobleizer · 2026-10-27
- Skyfall AI's SKY-001 world model turns sketches into CAD builds in 3.9s, vs 199s for GPT-6 Astra — rohanpaul_ai · 2026-10-27
- Physicists Let AI Pick Its Own Problem—It Wrote a Publishable Paper in 2 Hours — burny_tech · 2026-10-27
- Researcher flags unresolved data transparency issues in compositionality evals — najoungkim · 2026-10-27
- Separate Forms from Estimators: Why Steer by Difference of Means When You Can Use SGD? — johnhewtt · 2026-10-27