New Philosophy & Technology paper argues AI alignment should be principle-based, not preference-based
xuanalogue · x · 2026-11-17
Henrik D. Kugelberg and Tan Zhi-Xuan published a commentary in Philosophy & Technology, "Between Ideal and Non-Ideal: A Case for Principle-Based AI Alignment", responding to Pattison's non-ideal-theory defense of preference-based alignment.
- Core claim: preference-based alignment is not fully non-ideal — it carries consequentialist commitments, and human preferences over AI behavior already rely on normative principles.
- The authors argue that ideal theorizing about how to apply and adjudicate principles is sometimes crucial in hard cases with conflicting principles or harms (illustrated by "Competing Harms" and "Uncertain Harm" lab-break-in thought experiments).
- Determining alignment principles should also involve reflecting on the ideal roles AI should serve in a just society.
- Conclusion: principle-based alignment offers richer normative resources than purely consequentialist or harm-based justifications.
Published version on Springer; a paywall-free pre-print is also available.
More from AGI Musings
- Wayve's Marc Wulfmeier: truly useful household humanoids won't arrive before 2050 — m_wulfmeier · 2026-11-17
- Punitive AI disclosure mandates 'select for secrecy', economist warns — ipeirotis · 2026-11-17
- Sam Altman: trading agency for AI cures is a terrible sales pitch — haider1 · 2026-11-17
- The AI Cold War Is a Dark Forest: OpenAI Recorder Case and the Collapse of Lead Times — far__el · 2026-11-17
- When AI narrows choice: ChatGPT picked 2 of 31 ice cream flavors 96 times out of 100 — APPSO · 2026-11-17
- Security researcher: outages aren't vibe coding's fault — 'move fast, break things' is — GaryMarcus · 2026-11-17