Trading Off ASI Benefits: 1 Year Delay Buys 0.25% Lower Takeover Risk
DKokotajlo · x · 2026-08-12
Researcher RyanGreenblatt explores the tradeoffs of delaying ASI to mitigate misaligned AI takeover risks. He proposes a quantitative estimate: a reasonable aggregate of human preferences would be roughly indifferent between delaying the benefits of wildly superhuman AIs by a year and a 0.25% absolute reduction in takeover risk.
His rationale is based on the current human mortality rate of 0.75% per year. If one only cares about saving the lives of currently alive people in a rolling way, and assumes misaligned AI takeover carries similar mortality implications, a one-year delay yields roughly this amount of risk reduction. However, as a longtermist who cares deeply about future generations, he notes his personal preference would tolerate an even longer delay for added safety.
Related event: Former OpenAI Researcher Weighs Delaying ASI to Mitigate Risks(2 posts)→
More from AGI Musings
- Future Software: AI Agents Will Turn Code Into the New 'Kernel' — Fowe · 2026-08-12
- Big Lab Brain Drain: Why 1,000 'Neolabs' Will Spark a SF Renaissance — deedydas · 2026-08-12
- NBER Paper: GenAI Faces Oligopoly, Open Ecosystems Need a 'Rogue' Giant — soumitrashukla9 · 2026-08-12
- Palmer Luckey: Every Generation Fights New Tech, But AI Will End Scarcity — a16z · 2026-08-12
- The Enterprise AI Paradox: Why Renting Generic Models Is Unsustainable — heyshrutimishra · 2026-08-12
- Future AI Experience Will Be Like Early Clubhouse: Audio Rooms with AI Delegates — curious_vii · 2026-08-12