Paul Christiano: 1-in-3 chance of superintelligence within a year without slowdown
Don't Worry About the Vase (Zvi) · rss · 2026-11-20
Zvi's long-form breakdown of Dwarkesh Patel's podcast with Paul Christiano (plus three lectures) delivers Christiano's urgent public warning:
- Core claim: we may soon build AI far better than humans at almost everything; absent slowdown, 1/3 chance of superintelligence within a year and >10% loss-of-control risk.
- Specific worries: AI R&D itself is automating, compounding progress; frontier lab security can't protect model weights deployed on Azure/AWS; governance won't notice things going off the rails in time.
- Acceleration: METR and Anthropic internal data match benchmarks in showing AI progress roughly doubled over 6-12 months — faster than even self-described fast-takeoff people anticipated.
- Proposal: pause frontier algorithmic progress and prioritize security and alignment; pausing without fixing security risks weight theft and a worse race.
- Zvi also covers why an AI might want takeover, how it could happen, and sim-to-real transfer, concluding 'build the plane as we fly it' is the best available plan.
More from AGI Musings
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Anthropic mathematician claims he and Claude proved the Riemann Hypothesis, formalized in Lean — NinaDSchick · 2026-11-21
- The AI trade has been the compute buildout — and its end will get weird fast — matt_slotnick · 2026-11-21
- Human emotions are outdated atavism of bio hardware—can AI help us regulate them? — irinarish · 2026-11-21
- Walter Isaacson joins Google to discuss creativity and human judgment in the AI age — OnlineInference · 2026-11-21
- Ilya on Dwarkesh podcast: 'All this AI stuff... it's straight out of science fiction' — willdepue · 2026-11-21