Why RL is hard: specifying what you want is the bottleneck, says Andrew Carr
andrew_n_carr · x · 2026-10-11
Andrew Carr offers a terse take on why reinforcement learning is hard: the core difficulty is precisely expressing what you want — designing a good proxy reward for the behavior you actually want is genuinely hard, and models can easily cheat the reward, compounding the problem.
His conclusion: hard is good — the difficulty of proxy design and reward hacking is exactly what makes RL worth working on.
More from AGI Musings
- Spanish AI founder publishes 'Vita', an essay on AI, science and the limits of life — uribeetxebarria · 2026-10-11
- AI slop is a leadership failure first: studies of 10,000 leaders point to missing standards — DrKavner · 2026-10-11
- AI governance scholar warns the superintelligence race may be only ~8 months from its target — LuizaJarovsky · 2026-10-11
- We trust interpreters over raw information — and AI could help us trace claims instead — CyborgWriter · 2026-10-11
- Musk envisions a 'sentient sun': harnessing solar energy to build digital superintelligence — XFreeze · 2026-10-11
- Mathematician: genius worship fuels anxiety behind AI-era debates in math — arjunrajlab · 2026-10-11