AI safety debate: capability, power and loss of control are three different things
random_walker · x · 2026-11-21
A retweeted argument holds that AI safety discussions conflate three distinct dimensions: capability, power, and real-world impact. Proving the Riemann Hypothesis, for instance, wouldn't update priors on frontier AI's capability or on the capability-to-power transfer rate.
The attached table contrasts two safety stances:
- Superintelligence view: capability-to-power conversion is nearly inevitable (speed, persuasion); safety effort focuses on power-to-loss-of-control via model alignment; the term "intelligence" blurs capability and power.
- Normalist view: capability-to-power is the main locus of safety work, addressable via market incentives, regulation, control techniques, AI for defense, and anti-concentration policies, aided by slow diffusion; power-to-loss-of-control is essentially tautological for agentic systems.
Related event: Skeptical Take on AI Takeover Scenarios Sparks Safety Debate(3 posts)→
More from AGI Musings
- One-liner: AI could mechanomorphize humanity until replacing us counts as saving us — PierceLilholt · 2026-11-21
- Can AI Replace Programmers? Author Says Patching and Defending Systems Still Needs Humans — jessi_cata · 2026-11-21
- Phillip Isola: language is a camera — what's its color space and resolution? — phillip_isola · 2026-11-21
- Cambridge AI safety professor launches Firebell society with 'AI fire alarm picnic' in SF — DavidSKrueger · 2026-11-21
- AI circles abuzz: an AI-solved Riemann Hypothesis would be the final blow to the 'just a parrot' crowd — sebkrier · 2026-11-21
- "Everyone is slowly becoming an AI agent supervisor" — nikvassev · 2026-11-21