LLM moral choices lack stable rankings; averaging fakes human-like moral hierarchy
rohanpaul_ai · x · 2026-11-11
A University of Missouri-led paper finds that LLMs performing well on morality benchmarks doesn't mean they actually weigh competing values.
- Five LLMs (incl. GPT-5.2, Gemini, DeepSeek) judged pairs of moral violations, with choice probabilities read directly from next-token outputs.
- Reversing the order of the two options flipped the value ranking in 26 of 30 test blocks, showing choices lack a consistent ordering.
- Yet after averaging responses — a common practice — 4 of 5 models showed a clean, nearly identical hierarchy matching no individual test set: an aggregation artifact.
Takeaway: before trusting moral alignment scores, verify choices stay consistent per-prompt and across both answer orders, without averaging.
Related event: Study: LLMs Lack Stable Moral Value Rankings(2 posts)→
More from AGI Musings
- Why 'the next six months will be weird' AI predictions keep falling flat — sebkrier · 2026-11-11
- SimpleBench trend breaks: newest models beat trajectory on common sense — ChrisGPT · 2026-11-11
- AI safety community accused of reputation-driven opinion cascade on thin evidence — sebkrier · 2026-11-11
- Ex-OpenAI policy VP Miles Brundage: METR report redactions hurt public understanding less than claimed — Miles_Brundage · 2026-11-11
- 'Belief in CEV Is Belief in God': A Sharp Jab at Value Alignment Theory — mimi10v3 · 2026-11-11
- A 2035 Scenario: Trillions of AI Agents Do Knowledge Work While Humans Return to Hands-On Life — sebkrier · 2026-11-11