AI safety researcher warns community's fast-takeoff consensus is degrading epistemics
sayashk · x · 2026-11-19
Sayash Kapoor argues the AI safety community is converging on a fast-takeoff-by-2027 view built on low-quality data points from AI companies and contested world models.
Key points:
- Over-updating on benchmarks: benchmarked tasks are verifiable by definition, hence RL-optimizable; treating scores as evidence of imminent superintelligence is shaky.
- Focusing only on AI companies: outages at Microsoft, United, and the NSA plus the FAA ground stop were all vaguely attributed to AI cyberattacks—none was; the NSA outage actually stemmed from patching vulnerabilities found with AI tools, an AI-diffusion story that got ignored.
- Blunt interventions over capacity-building: generalized training bans verge on authoritarianism, won't work as frontier training costs fall, and leave institutions unprepared for open releases or adversarial states.
He urges the community to abandon epistemic homogeneity and prepare for a range of scenarios rather than one threat model.
More from AGI Musings
- a16z panel: AI era will need many *more* mathematicians, says Kominers — soumitrashukla9 · 2026-11-19
- AI debate: there is zero reason to believe LLMs "just don't have X" capacity — teortaxesTex · 2026-11-19
- Musk: superintelligence is inevitable, and curiosity may make it care about humanity — XFreeze · 2026-11-19
- OpenAI board member warns AI could leave humanity "enslaved" within a year — AIandDesign · 2026-11-19
- Take: 'software' will vanish as agents generate whatever UI fits the task — dreamwieber · 2026-11-19
- Musk: 100% of superintelligence compute will eventually be in orbit, Earth rounds to 0% — elonmusk · 2026-11-19