Arrow's Impossibility Theorem Invoked to Argue AI Alignment Is Mathematically Impossible
mikeflache · x · 2026-11-16
A repost highlighting a 1950 paper by Nobel laureate Kenneth Arrow to argue AI alignment is mathematically impossible:
- Arrow asked whether a perfectly fair voting system can aggregate group preferences, under three axioms: unanimity (if everyone prefers A over B, A wins); independence of irrelevant alternatives (adding option C doesn't flip A vs B); and no dictator.
- His proof shows any system satisfying two axioms must violate the third — no perfect aggregation mechanism exists.
- The post extends this social choice result to AI value alignment: if even human group preferences can't be consistently aggregated, aligning plural human values into a single model objective faces structural barriers.
More from AGI Musings
- AI researcher says his job is now almost fully automatable with Claude Code — jonkhler · 2026-11-16
- Bank AI ROI: 80% Say It Works, Only 1% Can Prove It, Turrin Newsletter Finds — mikeflache · 2026-11-16
- AI detection reveals many contemporary fiction professionals don't read or write — Afinetheorem · 2026-11-16
- dair-ai's top AI papers of the week: s3, AutoRefine, agent scaling — dair_ai · 2026-11-16
- Nina Schick: keeping AI out of China is impossible — the US must just build the best AI — NinaDSchick · 2026-11-16
- Frontier lab researcher: 'We don't know how to reliably instil values into AIs' — scientific consensus — wfithian · 2026-11-16