iamtrask: AI risk is best summarized as concentration of power, not misalignment
iamtrask · x · 2026-09-20
OpenAI researcher iamtrask posted a thread arguing that concentration of power is the best summary of AI risk: AI can hardly do harm before it has accumulated enough power, and RSI is essentially a feedback loop where concentrated power begets more concentration.
Two contrarian takeaways:
- Alignment only matters to the degree power is concentrated — if everyone's power rises equally, any particular AI's values don't matter much.
- The AI industry's biggest mistake is assuming (or wanting to believe) that maximizing AI's benefits requires concentrating power.
Responding to the counterexample of narrow superintelligence (self-driving cars, AlphaGo, AlphaFold), he says we could simply keep doing narrow AI, but companies face a perverse incentive: they also want to concentrate power, so they won't legitimize this narrative.
More from AGI Musings
- Oxford DPhil Thesis by Andrew Trask Argues Maximal AI Capability Runs Against Centralization — iamtrask · 2026-09-20
- AI slop grenades come from people who can't tell what good looks like — nickbaumann_ · 2026-09-20
- Researcher pushes back on denying reproducible results like reward hacking over tribal politics — burny_tech · 2026-09-20
- iamtrask: parallel contradictory strategies require defecting from prestige to legal fights — iamtrask · 2026-09-20
- iamtrask: AI safety community avoids copyright alliance to protect its vendor-lobbying strategy — iamtrask · 2026-09-20
- Embedded evaluator commitments by OpenAI and Anthropic are welcome but insufficient, argues Daniel Tan — MariusHobbhahn · 2026-09-20