A kill switch is not a safety architecture: preparing for AI smarter than humans
williamtp · x · 2026-09-19
A systematic essay on superintelligence risk (with a live SkyNews interview):
- For all of human history, the most intelligent beings we've encountered have been other humans; if AI keeps improving, that could change — and we have no experience sharing the world with something far smarter
- Uncertainty cuts both ways: extraordinary benefits possible, but inability to describe every danger isn't evidence there's nothing to worry about
- A kill switch is a sensible precaution but not a safety architecture — like a factory's red stop button, real safety comes from engineered safeguards throughout the system: layers of control and verification
- The unreliability driving safety concerns also keeps businesses from fully automating key processes: "trillions of dollars are trapped above this trust ceiling"
- Efforts to make AI trustworthy should match efforts to make it capable
More from AGI Musings
- Karpathy thinks in model generations: 2 years ahead is 8 generations away — chhaviyadav_ · 2026-09-19
- Musk predicts AI will double US GDP growth to ~4% next year — JosephJacks_ · 2026-09-19
- Philosopher RY Chappell: ordinary discount rates break longtermist moral math — AaronBergman18 · 2026-09-19
- Kevin Roose's final NYT column: the AI worry moment makes him more optimistic — kevinroose · 2026-09-19
- Kai-Fu Lee: CEOs can no longer delegate AI transformation — they must own it — kaifulee · 2026-09-19
- Hinton renews warning: AI will spiral out of control unless frontier labs slow down — Hesamation · 2026-09-19