New paper: revealed preferences alone aren't enough for AI alignment
joshgans · x · 2026-11-14
Economists Joshua Gans and Richard Holden released a new paper, "Are Revealed Preferences All You Need for AI Alignment?", challenging a popular CS assumption: that observing someone's choices suffices to train an aligned AI.
Key points:
- They test the claim in the most favorable setting imaginable—fully observing a person's revealed preferences over the space of beliefs—and still show it's insufficient.
- The reason: when an AI is asked to gather information, it chooses the experiments, and those experiments generate the beliefs the AI relies on for decisions. With only revealed preferences, its choice of experiments diverges from what a person would actually choose, compounding misalignment in downstream actions.
- The paper characterizes what drives this divergence, shows how to assess misalignment potential from decision contexts, and discusses remedies.
An explainer video by Holden accompanies the paper.
More from AGI Musings
- 'Hard Pill to Swallow: AI Is Pay to Win' — Tweet Sparks Pricing Debate — prasenx · 2026-11-14
- Fed Governor Barr: AI Could Boost Productivity While Widening US Inequality — robseamans · 2026-11-14
- Buyer spending $10M/yr on legal fees says AI will do 90% of M&A legal work within 5 years — jkubicki · 2026-11-14
- Dev pushes back on RSI hype: frontier labs just have 'infinite tokens' and use AI for everything — dyn___ · 2026-11-14
- Alignment researcher: 'solving alignment' in the current paradigm is impossible — j_foerst · 2026-11-14
- Rapid Gemini releases at the frontier spark super-exponential AGI optimism — iruletheworldmo · 2026-11-14