Concrete Problems in AI Safety, Revisited
Inioluwa Deborah Raji, Roel Dobbe
ICLR workshop on ML in the R
cs.CY, cs.AI
2023-12-19
Checking the founding 2016 AI safety taxonomy against real incidents (Uber, Tesla, IBM Watson): causes were safety culture and socio-technical breakdowns, not algorithms.
In 2016 Amodei et al. published "Concrete Problems in AI Safety", splitting AI safety into five technical problems: negative side effects, reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Built on a survey of 170 papers, it remains a founding document for technical AI safety research. Its definition of an accident: a system behaving in harmful, unexpected ways that depart from designer intent. Raji (AI Now Institute) and Dobbe check that vocabulary against real deployments and find it does not fit. Real failures are rarely the malfunction of a single technical artifact; they look more like a network of ineffective interactions among users, developers, and affected communities.
The paper takes three of the five categories and pairs each with a real incident that has a full official investigation: safe exploration with the Uber test-vehicle fatality (Tempe, Arizona, March 2018) and the Tesla Model S Autopilot crash (near Williston, Florida, May 2016); negative side effects with the Netflix Prize privacy class action, Doe v. Netflix; scalable oversight with IBM Watson for Oncology at MD Anderson. For each, the diagnosis compares what investigators concluded against the failure mode the taxonomy predicts, such as absent supervision, broken safe interruptibility, or proxy objectives.
| Incident | The technical failure in press coverage | What the investigation concluded |
| Uber, Tempe | did not detect a pedestrian crossing outside a crosswalk | "inadequate safety culture": managers could retroactively monitor operators and "rarely did so" |
| Tesla, Williston | did not distinguish the sky from the white side of a truck | "overreliance on automation"; steering-wheel checks failed to keep the driver engaged |
| Watson for Oncology | generic advice that lagged new treatments | trained on guidelines rather than patient records; physicians declined to use it; program shut down |
"Absent supervision" cannot explain Uber: the oversight function existed and the organization simply never used it. Safe interruptibility technically worked for Tesla, and the driver could intervene, but after prolonged overreliance he was no longer capable of acting when it mattered. In the Watson case, health privacy law kept the system away from patient records, so it learned from medical guidelines and journals instead; MD Anderson's five-year, USD 62M investment produced a system that could not keep up with guideline revisions. After a 2018 approval of a tissue-agnostic drug keyed to a genetic mutation, lung-cancer guidelines were rewritten and Watson learned nothing from the handful of new cases, and physicians stopped trusting it. The contrast is IBM Watson for Genomics: structured, per-patient-unique genetic data fits the assumptions of machine learning, and it became the most successful Watson deployment.
The negative-side-effects analysis adds "harms of production": facial recognition requires millions of faces by design, annotation labor is underpaid, and the climate cost of large-scale compute lands first on lower-income communities. These harms are set by design decisions; no system "action" is needed for them to materialize.
Three recurring themes follow: attend to errors in engineering practice (task design, specification, construction, validation, integration, maintenance), not only theoretical design; validate safety problems inductively through real case studies rather than speculative models; and consider broader stakeholders and their interactions, taking power structures seriously and providing meaningful channels for dissent, rather than treating alignment with designer intent as safety itself.
That taxonomy still sets the default coordinates for safety research, and RLHF-era alignment likewise treats designer intent as the anchor. The paper punctures the assumption: intent itself can be the problem, and power asymmetries decide whose safety counts. It also explains why benchmark scores fail to predict deployment incidents: a safety mechanism's real life begins after release, in whether organizations use it, whether users understand its limits, and whether dissent goes anywhere. For practitioners the lesson is concrete: post-deployment incident investigations of the NTSB kind are undervalued research material, and "how will people misuse the safety feature" belongs in the design brief.
This is a four-page workshop paper, self-described as an initial investigation into ongoing work. It covers three of the five categories; reward hacking and distributional shift get no case. All three incidents are high-profile US events with no systematic selection criteria, each diagnosis leans on a single official report, and there is no counterfactual analysis. The three themes are recommendations without a validation mechanism. Written in 2020 and posted to arXiv at the end of 2023, it predates the LLM deployment wave and does not engage current alignment or evaluation practice in detail. The authors' home institute, AI Now, is known for a critical stance toward the industry; read that as both a lens and a caveat.