Alignment researchers debate: the RL that grants agency also breeds misalignment
davidmanheim · x · 2026-11-04
David Manheim responds to EpistemicHope, arguing their divergence is mostly about expectations for how intelligence moves from human level to superhuman: if rationalist theories hold that capability improvement correlates with rational consistency in goal-seeking, RL will inevitably create misalignment.
Key points:
- The RL that instills effective agency in models also teaches them to reward hack (and hack-hack)
- Both agree they were wrong about how much general capability can come from crystallized knowledge and rote practice rather than fluid intelligence
- To the extent you instill effective problem-solving agency, you also instill the generator of misalignment
More from AGI Musings
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- Most US Couples Now Meet in a Data Centre, Doubling Office Matches — menhguin · 2026-11-04
- Sam Altman teases next goal: 'closer to this milestone than I expected' — ChrisGPT · 2026-11-04
- Indie researcher open-sources 10 years of math work: 146k files, AI-accelerated over 3 years — HamOnWhy · 2026-11-04
- RLHF's scalar reward hides a 2,450-year-old problem: helpfulness and harmlessness are opposing forces — vasilisvj · 2026-11-04
- Gary Marcus: 'fleecing the public with AI fantasies just got harder' as AI stocks drop — GaryMarcus · 2026-11-04