In defense of AI safety's talent: 'these people can't do math' is not a serious argument
algekalipso · x · 2026-09-17
Pushing back on the emerging rhetoric that EA and AI safety communities are full of people who 'don't know how AI works,' the author argues the field has an unusually dense concentration of cognitive outliers — IMO medalists, elite mathematicians, top-lab engineers — rivaling the Manhattan Project.
Key examples: Nate Soares went from Google engineer to MIRI research fellow in about a year, self-teaching category theory, model theory and provability logic, famously constructing massive proofs with 'the type checker in his head'; Evan Hubinger vouches for it. And Hinton, 2024 Nobel laureate and neural network pioneer, left Google to warn about loss-of-control risk.
The author agrees loss of control is serious, but says the community's real blind spot is consciousness: phenomenal binding, functionalist assumptions, and whether the agent or its subsystems are the relevant moral patients. Disagree with their conclusions if you like — 'they're uninformed' is a terrible model of what's happening.
More from AGI Musings
- Chinese AI labs run 6-9 months behind but are far more product-focused, researcher says — Afinetheorem · 2026-09-17
- Rereading Wait But Why's 'AI Revolution' now that AGI debates went mainstream — cnakazawa · 2026-09-17
- Aeon essay: computism explains our datafied lives — but who tastes the cherries? — round · 2026-09-17
- AI researcher reverses stance: most lawyers likely replaced by AI within years — LuizaJarovsky · 2026-09-17
- DeepMind researchers propose pragmatic framework for AI personhood as a bundle of obligations — jzl86 · 2026-09-17
- multimodalart on fly brain experiments: small neural networks are underrated — multimodalart · 2026-09-17