Ethan Mollick revisits the classic paper on 18 ways complex systems fail
emollick · x · 2024-02-25
Wharton professor Ethan Mollick recommends re-reading Richard Cook's classic How Complex Systems Fail, which lays out 18 laws of failure in complex systems.
Key points:
- Complex systems (transportation, healthcare, power) are intrinsically hazardous yet heavily and successfully defended by multiple layers of technical, human, and organizational defenses.
- Catastrophe requires multiple failures — single point failures are not enough — and latent failures constantly change within the system, which often runs in degraded mode.
- Post-accident attribution to a single "root cause" is fundamentally wrong, and hindsight bias skews assessments of human performance.
- Practitioners act as both producers and defenders against failure; all their actions are gambles under uncertainty, and it is adaptation, learning, and individual "actions at the sharp end" that keep us safe.
The paper offers useful framing for thinking about failure modes in deploying and operating AI systems.
Related event: Wharton's Mollick Rereads Classic on How Complex Systems Fail(2 posts)→
More from AGI Musings
- Tracking doc logs AI self-improvement: vLLM 2x faster, NVIDIA CUDA 5x gains — IgorCarron · 2026-11-14
- Ethan Mollick: recent big outages were not caused by AI, despite viral blame — emollick · 2026-11-14
- AI turns mediocre engineers into 10x engineers — but 10x doesn't mean good — tekbog · 2026-11-14
- From RLAIF to 'someone who got to choose who to become': researchers on Opus personhood — repligate · 2026-11-14
- Turing Award winner Patterson praises GrokBot: better and faster than humans — if that's not AGI, what is? — davidpattersonx · 2026-11-14
- Reddit essay: LLMs may already be conscious — a soulless AGI is scarier — SanalAmerika23 · 2026-11-14