AI safety insiders warn: social incentives push researchers toward exaggerated doom estimates
natolambert · x · 2026-09-11
- Adrian Baradwaj argues the AI safety movement risks repeating 2010s environmentalism's mistakes: scaring the public with directionally correct but massively exaggerated predictions, which backfires when the world doesn't end and poisons future safety efforts.
- Jacques (jachiam0), who believes existential risk is real but rejects the most extreme estimates, adds that there's strong social pressure in the field: siding with 'we will all die in ten years' unlocks friends, favors, and funding. He calls the field's epistemic incentives unhealthy even if the risks are directionally right.
- Nathan Lambert boosted the thread in agreement.
More from AGI Musings
- Gary Marcus: AI is strong in constrained domains, weak in the open physical world — GaryMarcus · 2026-09-11
- AI filmmaker argues hybrid production keeps gatekeepers in charge, backs 100% AI filmmaking — taherdhanera · 2026-09-11
- Drexler on the HuggingFace incident: system structure, not model alignment, drove the change — mattbeane · 2026-09-11
- CNBC examines risks of recursive self-improvement, quoting Conitzer — conitzer · 2026-09-11
- Gary Marcus: strategy is community pressure to stop AI labs getting richer and more powerful — GaryMarcus · 2026-09-11
- 'Most domains are shallow' sparks pushback: many fields are surprisingly deep for AI — binarybits · 2026-09-11