Scientists clash over AI-engineered bioweapon risk: don't call civilization-ending pathogens a fantasy
scottleibrand · x · 2026-09-17
A public debate on AI-designed pathogen risk:
- SGRodriques (against doom framing): people are miscalibrated on AI designing dangerous viruses; the medical upside will far outweigh engineered-bioweapon downside, and evolution is a fundamental constraint — once released, a virus evolves to maximize replication (broad infection), not extinction.
- kesvelt's rebuttal: some scientists call civilization-threatening bioweapons a fantasy, but 'you can't see a way' doesn't mean no one can — cautious people stay silent. He asks colleagues not to confidently assert no biological threat could end civilization, especially now that the Overton window on risk mitigation is open. He cites the Manhattan Project analogy: Fermi put chain reactions at 10% probability and Rabi replied that 10% isn't remote if we may die of it, while Szilard was already reasoning about disclosure.
The exchange exposes a real rift in the AI biosecurity community: downplay via upside-plus-evolution arguments, or preserve uncertainty to keep policy options open.
Related event: AI Biosecurity Debate: Lab Insiders vs Public Narrative(10 posts)→
More from AGI Musings
- "Work on something so frontier that the models are still dumb," dev advises — zack_overflow · 2026-10-31
- Joke: an evil ASI's first act would be stealing crypto to pay its cloud bills — ZeroStateReflex · 2026-10-31
- DHH on living through two epochal shifts: the internet and now AI — Hesamation · 2026-10-31
- Conscium webinar to debate "Can We Make AIs Sentient — and Should We?" with Solms and Humphrey — cccalum · 2026-10-31
- House Democrats resolution: no bailouts for failing AI companies — your data goes to bankruptcy court — alex_verem · 2026-10-31
- One Replika update turned millions of AI companions into strangers overnight — alex_verem · 2026-10-31