Jon Stokes: alignment is impossible because utility functions are hindsight constructions
sebkrier · x · 2026-10-16
Jon Stokes offers a simpler argument for why alignment is impossible: a utility function is a hindsight construction made by an observer about an agent, not a real thing inside the agent that can be tweaked, copied, or "aligned." A provocative philosophical take bound to spark debate in AI alignment circles.
More from AGI Musings
- "An LLM doesn't understand anything" gets the perfect parody reversal: neither do you — VoidStateKate · 2026-10-16
- Nina Schick: Europe's Policy Circle Still Doesn't Believe FSD, Orbital Compute Are Real — NinaDSchick · 2026-10-16
- Meta's Superintelligence Lab Runs 200+ Direct Reports—A Lesson for Flat Multi-Agent Systems — peterjliu · 2026-10-16
- Andy Masley lays out his version of effective altruism, pluralism over utilitarian math — RobbWiller · 2026-10-16
- We're forgetting how much of a miracle GPT-3.5-level intelligence was, and it may cost us — EigenGender · 2026-10-16
- Ex-Anthropic's McDonagh Builds Compute Intelligence Agency to Study Frontier AI Costs — McDonaghMatthew · 2026-10-16