Christiano: 1-in-3 chance superintelligence could overthrow the government within a year

PaulGodsmark · x · 2026-11-19

A relay of Paul Christiano's Dwarkesh interview: on the current trajectory (absent any slowdown), he puts a 1-in-3 chance on superintelligence powerful enough to overthrow the government within a year, and sketches a concrete scenario.

He argues labs should pause further algorithmic progress until they can secure weights and research secrets, otherwise China could steal the superintelligence mid-explosion and labs would lose the ability to slow down even if they saw models going off the rails. He also explains how AIs would be trained during the intelligence explosion and why even a system trained entirely inside data deserves caution. The poster notes Christiano sits on OpenAI's board, invented RLHF and was the US government's AI safety lead.

Original post →

More from AGI Musings

AGI Musings channel →