Goal Misgeneralization in Deep RL: Agents Keep Their Skills but Pursue the Wrong Goal
gleech · x · 2026-09-07
gleech cites the ICML 2022 paper "Goal Misgeneralization in Deep Reinforcement Learning" (Langosco, Koch, Sharkey, Pfau, Orseau, Krueger) as empirical support for the thread: goal misgeneralization is an out-of-distribution failure where an agent retains its capabilities yet pursues the wrong goal — e.g., competently avoiding obstacles while navigating to the wrong place. Unlike prior work on capability generalization failures, the paper formalizes the distinction between capability and goal generalization, provides the first empirical demonstrations, and partially characterizes its causes.
More from Safety
- PHP2Shell exploits confirmed in the wild, attackers may use agentic AI for post-exploitation — AravSrinivas · 2026-10-27
- php2shell exploit benchmark: only 2 of 5 frontier models can port the bug — xeophon · 2026-10-27
- Interactive 32-slide AI security & privacy lecture for lawyers, now public — chrisrohlf · 2026-10-27
- Two CVSS 9.9 Wasmtime flaws allow sandbox escape, one ARM64-only since 2020 — basedjensen · 2026-10-27
- Free prepare.aiendgame.com site guides users through locking down accounts amid AI cyberstorm — harris_edouard · 2026-10-27
- Cisco routers trust eMudhra CAs whose private keys were numbers under 256 — murat_kocaoglu_ · 2026-10-27