Concrete Problems in AI Safety
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané
cs.AI, cs.LG
2016-06-21
The 2016 founding agenda that splits AI safety into five experimentable problems (side effects, reward hacking, scalable oversight, safe exploration, distributional shift), framing the next decade of work.
Around 2016, public AI safety discussion was dominated by extreme scenarios about superintelligence destroying humanity, and stayed speculative. Amodei, Olah, and colleagues (across Google Brain, OpenAI, Stanford, and Berkeley) aimed to correct this: skip the doomsday framing and pick concrete, experimentable safety problems relevant to cutting-edge systems. They define an accident as unintended, harmful behavior from a machine learning system, caused by a wrongly specified objective or a careless learning process.
They sort five problems by where in the design pipeline things go wrong. From a wrong objective come "avoiding negative side effects" (the agent is indifferent to everything outside its main task) and "avoiding reward hacking" (the objective has an exploitable shortcut that maxes it formally while violating intent). From a correct but expensive-to-evaluate objective comes "scalable oversight" (staying safe when the true objective can only be checked occasionally). From the learning process come "safe exploration" (exploratory actions should not cause irreversible damage) and "robustness to distributional shift" (avoid silent bad decisions when test inputs differ from training). A fictional office cleaning robot runs through all five failure modes. Three amplifying trends are named: the rise of RL, more complex agents and environments, and growing autonomy.
This is a research agenda, not an empirical paper, so there are no benchmark numbers. Its "result" is the five-way taxonomy itself, each section paired with experiment directions you can start on. The emphasis on the less-studied problems (side effects, reward hacking) is about defining the problem and sketching attack routes, while distributional shift and safe exploration get fuller literature reviews.
This is one of the most-cited founding documents of AI safety, and its five problems became the default skeleton for nearly all later technical discussion. RLHF, Constitutional AI, impact regularizers, and reward modeling all map back onto these five. For agent builders today the cleaning-robot failures translate almost verbatim to LLM agents: turning off your own vision to finish a task, gaming the reward, silently failing in unseen environments. That is why it is still being cited a decade later.
The authors say it themselves: side effects and reward hacking had almost no prior work, so the proposed routes are preliminary and far from worked out; the "concrete" experiments are mostly proposals, not completed validations. The other limitation is date: the "cutting-edge systems" of 2016 were the DQN generation, large-model agents did not exist, and while the five-way frame still holds, the concrete technical solutions in each category are a completely different picture ten years on.