Alignment thought experiment: treat utility functions as PII under a Kantian veil
jessi_cata · x · 2026-11-07
jessicata proposes an alignment thought experiment: if an agent's utility function counts as personally identifying information, could it boot up reasoning with no identity info, decide which logical commitments to make behind a Kantian veil of ignorance, and only learn its utility function at a later stage?
She flags the paradox herself: before knowing your utility function, how do you judge which pre-commitments are good? And if identity info trickles in gradually, what prior should early stages hold? An open alignment-philosophy thread with no resolution.
More from AGI Musings
- Companies are quietly rehiring roles they cut in the name of AI, suggesting AI wasn't the real reason — DavidLinthicum · 2026-11-07
- Researcher: AI community's outrage drops once LLMs master the thing it criticized — RexDouglass · 2026-11-07
- Researcher: the mob cared about citations until LLMs got good at them, then stopped instantly — RexDouglass · 2026-11-07
- AI Safety Researcher Vows to Stay, Calls on OpenAI and Anthropic to Stop RSI — trevposts · 2026-11-07
- AI Safety Researchers List Concrete Ways to Contribute Beyond Fellowships Like MATS — AndyMasley · 2026-11-07
- Garrison Lovely: data center moratoriums won't slow AI, chip export controls would — GarrisonLovely · 2026-11-07