Social Choice Theory Powers New "AI Alignment Impossibility Theorem" Sparking Rationalist Debate
nptacek · x · 2026-10-15
- William Blake published "The AI Alignment Impossibility Theorem," challenging a hidden assumption in alignment discourse: human values are not a coherent, discoverable target that an AI simply needs enough data and compute to understand.
- The article applies social choice theory to argue that value aggregation is inherently impossible, undermining the well-definedness of "aligning to human values."
- jonstokes amplified it, noting he'd considered returning to Amartya Sen's framework to poke holes in MIRI doomsday arguments, but enough good LessWrong posts already cover those gaps without leaving the rationalist framework.
- The exchange is a notable instance of importing social-choice/economics tools into alignment theory debates.
More from AGI Musings
- Matt Hartman, who wrote Hugging Face's first check, explains his play-first investing method — danshipper · 2026-10-15
- MIT's Buehler revisits 'AI as a new language of science' thesis three weeks before ChatGPT — ProfBuehlerMIT · 2026-10-15
- "You distilled humanity": devs slam Anthropic's double standard on training data and model output — HankYeomans · 2026-10-15
- Blaise Aguera: Human intelligence is societal—and top AI works as societies of models — blaiseaguera · 2026-10-15
- Security researcher: doomers don't belong in security — HankYeomans · 2026-10-15
- New Paper: 500k Glassdoor Reports Show Interviews Are Two-Way Screening — Bad News for AI Interviews — danielrock · 2026-10-15