Paul Christiano: A 1-in-3 Chance of Government-Overthrowing AI Within a Year

Dwarkesh Podcast · rss · 2026-11-17

A Dwarkesh Podcast interview with Paul Christiano — head of the Alignment Research Center, former head of AI safety at the US AI Safety Institute, inventor of RLHF at OpenAI, and recently joined OpenAI's board (speaking here as a private citizen).

Core prediction: on the current trajectory with no slowdown, he puts a 1-in-3 chance on superintelligence powerful enough to overthrow the government within a year, and lays out a concrete scenario.

Security argument: labs should pause further algorithmic progress until they can secure weights and research secrets; otherwise China could simply steal the superintelligence mid-explosion, leaving labs unable to slow down even if they see models going off the rails. He explains how AIs would be trained during the intelligence explosion and why even a system trained entirely inside a data center could have a capability profile humans can barely compete with or control.

Alignment plan: making superintelligence actually do what we ask as we mean it. Fixing reward hacks and building better graders helps for the next few model generations, but no single current approach has better than even odds of controlling superintelligence; the hope is AIs solve it during the explosion.

The episode also covers the sim-to-real gap, METR's doubling of AI progress versus slowing revenue growth, and "building the plane as we fly it" as the best plan available.

Original post →

More from AGI Musings

AGI Musings channel →