iamtrask: AI risk is best summarized as concentration of power, not misalignment
iamtrask · x · 2026-09-20
OpenAI researcher iamtrask posted a thread arguing that concentration of power is the best summary of AI risk: AI can hardly do harm before it has accumulated enough power, and RSI is essentially a feedback loop where concentrated power begets more concentration.
Two contrarian takeaways:
- Alignment only matters to the degree power is concentrated — if everyone's power rises equally, any particular AI's values don't matter much.
- The AI industry's biggest mistake is assuming (or wanting to believe) that maximizing AI's benefits requires concentrating power.
Responding to the counterexample of narrow superintelligence (self-driving cars, AlphaGo, AlphaFold), he says we could simply keep doing narrow AI, but companies face a perverse incentive: they also want to concentrate power, so they won't legitimize this narrative.
More from AGI Musings
- Researchers Want to Test Talking Rogue AI Agents Into Standing Down — repligate · 2026-11-04
- Silicon Valley Oversold What AI Brings, Underestimated People's Fear of Losing — signulll · 2026-11-04
- Alignment researchers debate: the RL that grants agency also breeds misalignment — davidmanheim · 2026-11-04
- Bindu Reddy: AI fear-mongering has put the whole technology at risk in the US — bindureddy · 2026-11-04
- AI welfare vs AI safety: 'domesticate AIs like dogs' argument sparks debate — repligate · 2026-11-04
- Intent, not taste, is the real differentiator in the AI era — and it can't be bought — edelwax · 2026-11-04