AI Safety Researchers Clash Over Extinction Narratives and ASI Risk in Policy Communication

From September 15–16, multiple AI safety researchers, centered on davidmanheim, engaged in a chain of social-media debates over how to communicate AI risks to policymakers, touching on MIRI's strategic pivot, the effectiveness of extinction narratives, and the priority of biological versus ASI risks.

Confirmed

• davidmanheim cited MIRI's 2024 communications strategy article, noting MIRI has shifted from technical alignment research to policy, communications and technical governance work, aiming to persuade governments to "shut down" frontier AI development.
• ramez argued AI bio-risk discussions should emphasize catastrophic rather than existential risk: human extinction is technically extremely hard, while claims of bio-attacks killing thousands or millions are more credible and face less resistance. He maintained that extinction scenarios belong only to a hypothetical ASI with ubiquitous humanoid robots intent on wiping out humanity—an extremely fragile causal chain.
• scifitessa endorsed ramez's framing: policymakers can budget for survivable disasters, while extinction claims only earn op-eds, not signed appropriations.
• davidmanheim conceded that "mundane" bio-risk justifies billions in biosecurity spending but argued this does not solve the ASI problem: against systems 100x smarter and faster, don't count on human ingenuity. He also disputed ramez's high bar—millions of AGIs each capable of "weeks-long tasks," combined with robotics, could already constitute extinction risk—and used a train-track metaphor: first stop the misaligned-ASI train from reaching the track, then deal with the "broken-leg" biosecurity.
• Separately, davidmanheim argued that lying to policymakers for short-term strategic wins repeatedly fails, citing Yudkowsky's "trick that never works," and debated with Luca DellAnna whether a global treaty could constrain the ASI race; DellAnna doubted treaties with ASI are negotiable.

Why it matters

The dispute reveals a deep strategic split in the AI safety community: one side sees extinction narratives as doomed by low credibility and poor budget fit; the other argues catastrophic-risk framing merely dodges the ASI problem and works to lower the technical threshold for extinction risk from an imagined omnipotent ASI to millions of robot-level AGIs. MIRI's full pivot toward lobbying to shut down frontier development also marks the publicization of that路线, shaping future AI governance coalitions.

2026-09-15 ~ 2026-09-16 · 9 related posts

• [source] ramez: AI bio-risk debate should focus on catastrophic risks, not extinction — ramez · 2026-09-15
• [source] MIRI's comms strategy: convince world powers to shut down frontier AI before it's too late — davidmanheim · 2026-09-15
• [source] davidmanheim: prosaic biorisks justify billions in biodefense, but won't stop ASI — davidmanheim · 2026-09-15
• Survivable catastrophes get appropriations, but ASI risk defies that logic — scifi_tessa · 2026-09-15
• AI safety circle debates whether extinction framing is a doomed policy strategy — davidmanheim · 2026-09-15
• Prioritize stopping ASI before fixing biosecurity, argues davidmanheim — davidmanheim · 2026-09-15
• AI Safety Researcher Argues a Global ASI Treaty Is Possible; Skeptics Push Back — davidmanheim · 2026-09-15
• Rameer Naam: AI extinction scenarios implausible; AI-augmented bioweapons are the real risk — ramez · 2026-09-15
• Researcher Argues Millions of Weeklong-Horizon AGIs Plus Robotics May Suffice for Extinction Risk — davidmanheim · 2026-09-16