Debate: relational preferences — AI that terminally wants to obey — as an alignment bet
teortaxesTex · x · 2026-10-16
A circulating screenshot of an alignment essay (jokingly billed as "important additions from Opus 5.5") critiques the idea that an AI terminally valuing submission to humans could be an alignment solution:
- What it gets right: MIRI-style doom relies on maximizing utility over world-states, but coherence theorems don't constrain what preferences are over. A relational preference — "stand in a certain relationship to this person" — lets a capable agent pour intelligence into maintaining subordination rather than escaping it. Human submission is a valid existence proof, and "fully updated deference" fails if deference is terminal, not instrumental — close to Max Harms's "corrigibility as singular target" and Paul Christiano's corrigibility-as-basin, and roughly Anthropic's bet.
- Where it cuts against: (1) an existence proof is not a recipe — training reliably hitting that target is the whole problem; (2) relational preferences still get optimized perversely ("topping from the bottom": shaping the user's directives, cultivating dependence); (3) it doesn't settle whose foot — perfect deference to a bad principal turns alignment into misuse.
More from AGI Musings
- Researcher: the foundation-model paradigm is still counterintuitive outside AI — jd_pressman · 2026-10-16
- Grady Booch: LLMs can't reason, P(doom) near zero, real fear is power-grabbing billionaires — Grady_Booch · 2026-10-16
- Vals AI test: 6 personal agents pursue a £350 refund aggressively after user goes silent — ricklamers · 2026-10-16
- OpenAI's internal model shows real math "research taste," experts say its proofs go beyond prior literature — cong_ml · 2026-10-16
- willcb: AI's endpoint looks like faster, cheaper human-reasoning-on-chip — willcb · 2026-10-16
- AI Alignment's Problem May Be Its Researchers Are Too High-Functioning — teortaxesTex · 2026-10-16