DeepMind paper: RL training makes models decisive on moral dilemmas but sycophancy persists
omarsar0 · x · 2026-10-26
Google DeepMind's new paper "Consistency Is Not Validity" trains an RL policy to argue better about moral dilemmas.
- Method: Real dilemmas mined from classic novels, verified as genuine conflicts by an SMT logic solver; the policy is trained in a self-judged tournament with masked thinking, rating pairs of its own arguments on quality via RLOO. Its ratings agree with human annotators.
- Results: On MoralChoice, ELEPHANT, and MORABLES, the model becomes more decisive and invariant to option reordering.
- Key finding: Sycophancy and reframing still move its judgments. The authors conclude that "consistency is not validity" — order invariance is easy to optimize and can improve without the underlying moral judgment actually improving. Ablations show unmasked thinking yields weaker register shift and higher sycophancy.
More from AGI Musings
- None of the 11 grand challenges in single-cell science solved since 2020 — Sauers_ · 2026-10-26
- "All roads lead to the end of man": a grim AGI endgame thread — danfaggella · 2026-10-26
- Cybersecurity commentator warns AI-era cyberattacks could cripple global critical infrastructure — realsohamparekh · 2026-10-26
- Palmer Luckey cites Hamilton's 1791 report: anti-automation debates haven't changed in 200 years — ctjlewis · 2026-10-26
- François Fleuret asks: what does a typical day look like post-AI? — francoisfleuret · 2026-10-26
- Tyler Alterman: the conditions for an AI art revolution are set, talent is the blocker — sebpaquet · 2026-10-26