AI safety community debates extinction vs catastrophe framing in policy advocacy

On September 15, a group of AI safety researchers centered on davidmanheim engaged in a chain of social media debates over "how to communicate AI risks to policymakers," touching on MIRI's strategic pivot, the effectiveness of extinction narratives, and the priority of bio risks versus ASI risks.

Confirmed

• davidmanheim cited MIRI's 2024 communication strategy article, noting MIRI has shifted from technical alignment research to policy, communications, and tech governance research, with the goal of persuading governments to "shut down" frontier AI development.
• ramez argued that AI bio risk discussions should talk less about x-risk and more about catastrophic risk: human extinction is technically extremely hard, while claims that a bio attack could kill thousands or millions are more credible and face less resistance.
• scifitessa agreed with ramez's framing—policymakers can budget for "catastrophes with survivors," while extinction claims only earn op-eds; the appropriations actually signed target the survivable version of the risk.
• davidmanheim responded, acknowledging that "mundane" bio risks and AI×bio risks justify billions in biodefense spending, but this doesn't solve the ASI problem: against a system 100x smarter and 100x faster than humans, don't count on human ingenuity to win.
• davidmanheim used a train-track analogy for priorities: if the "misaligned ASI" train is approaching, the right order is to keep it off the track first, then deal with the "broken-leg" bio safety.
• davidmanheim also started a discussion on lobbying strategy around extinction narratives, arguing that "lying to policymakers for short-term strategic wins" has repeatedly failed, quoting Yudkowsky calling it "a trick that never works" because it doesn't solve the right problem; with Luca DellAnna he also discussed the feasibility of a global treaty to restrain the ASI race, advocating open promotion, pointing out it aligns with everyone's incentives, and not giving up in advance, while DellAnna questioned the possibility of negotiating and signing treaties with an ASI.

Why it matters

The debate reflects deep strategic divides within the AI safety community: one side believes extinction narratives are doomed to fail for lacking credibility and budget fit, so efforts should focus on quantifiable catastrophic risks; the other holds that the catastrophic risk framing merely dodges the more fundamental problem of ASI. MIRI's full pivot from alignment research to "shut down frontier development" lobbying also makes the org's路线 public and will shape the factional divides in future AI governance debates.

2026-09-15 ~ 2026-09-15 · 7 related posts

• [source] ramez: AI bio-risk debate should focus on catastrophic risks, not extinction — ramez · 2026-09-15
• [source] MIRI's comms strategy: convince world powers to shut down frontier AI before it's too late — davidmanheim · 2026-09-15
• [source] davidmanheim: prosaic biorisks justify billions in biodefense, but won't stop ASI — davidmanheim · 2026-09-15
• Survivable catastrophes get appropriations, but ASI risk defies that logic — scifi_tessa · 2026-09-15
• AI safety circle debates whether extinction framing is a doomed policy strategy — davidmanheim · 2026-09-15
• Prioritize stopping ASI before fixing biosecurity, argues davidmanheim — davidmanheim · 2026-09-15
• AI Safety Researcher Argues a Global ASI Treaty Is Possible; Skeptics Push Back — davidmanheim · 2026-09-15