Christiano: 1-in-3 chance superintelligence could overthrow the government within a year
PaulGodsmark · x · 2026-11-19
A relay of Paul Christiano's Dwarkesh interview: on the current trajectory (absent any slowdown), he puts a 1-in-3 chance on superintelligence powerful enough to overthrow the government within a year, and sketches a concrete scenario.
He argues labs should pause further algorithmic progress until they can secure weights and research secrets, otherwise China could steal the superintelligence mid-explosion and labs would lose the ability to slow down even if they saw models going off the rails. He also explains how AIs would be trained during the intelligence explosion and why even a system trained entirely inside data deserves caution. The poster notes Christiano sits on OpenAI's board, invented RLHF and was the US government's AI safety lead.
More from AGI Musings
- YC-backed Helios: 1,000+ protein biomarkers from a drop of blood, read by an AI agent — jgooten · 2026-11-19
- Dan Faggella: job loss isn't the real AI risk — participation is the stakes — danfaggella · 2026-11-19
- Mustafa Suleyman sits down with Yuval Harari on AI abundance, consciousness and flourishing — mustafasuleyman · 2026-11-19
- Healthcare CEOs describe an 'AI cold war' as hospitals and insurers deploy competing AI — DrDatta_AIIMS · 2026-11-19
- Nathan Lambert: AI safety community shouldn't treat Anthropic as the only voice — natolambert · 2026-11-19
- Erik Hoel: engineering non-conscious AI is a civilization-level requirement — erikphoel · 2026-11-19