Questioning AI Safety's Capability Extrapolation
AdrienLE · x · 2026-07-12
The author shared a commentary, noting a recurring pattern in "doomer" AI safety research:
- First, treating a human "talent" (like persuasion, discernment, or charisma) as a single scalar;
- Assuming this ability is primarily determined by intrinsic individual factors, rather than context or environment;
- Hypothesizing that this scalar can be infinitely scaled up;
- Using this to extrapolate the future of AI.
They give the example that concepts like "superhuman persuasiveness" might not hold up, because "persuasiveness" may not be a stable human talent, but rather an attribute retroactively credited to an individual during a social process.
More from AGI Musings
- Researcher quits Anthropic, says OpenAI and Anthropic are gambling lives racing to self-improving superintelligence — davidmanheim · 2026-09-11
- Misquoted: Anthropic Staff Warned of Double-Digit Extinction Risk by 2030, Not Dismissed It — davidmanheim · 2026-09-11
- Economist Ben Moll: You Can Model Anthropic's 15% AI GDP Growth, But It Won't Happen — sebkrier · 2026-09-11
- Cohere Labs launches interactive tool mapping which tasks of 178 occupations AI can automate — Cohere_Labs · 2026-09-11
- AI researcher on SkyNews flags concerns over inequality, power and criminal misuse — schwarzjn_ · 2026-09-11
- VC compares AI doom rhetoric to pandemic-era fear messaging — StewartalsopIII · 2026-09-11