Do Misaligned Mesa-Optimizers Actually Exist? Researchers Weigh the Evidence
dioscuri · x · 2026-09-27
Responding to Herbie Bradley's question about real examples of mesa-optimisation, dioscuri notes we have evidence of learned planning (e.g. the Sokoban paper) but no clear-cut case of a misaligned AI mesa-optimizer; still, since evolution produced biological planners whose goals diverge from reproductive fitness, there's some reason to take the possibility seriously in AI.
The exchange is part of a broader alignment debate about the strength of evidence for inner-misalignment risk, later countered by Quintin Pope's argument that the evolution analogy is unhelpful.
More from Safety
- Cambridge Digital Minds hires postdocs on AI welfare and governance — davidmanheim · 2026-11-16
- Pangram's 750-word false-positive bounty went unclaimed for 5 days — nobody wants to gamble $1,000 — ctjlewis · 2026-11-16
- Researcher reproduces Pangram's FP rate but shows a long epic poem can evade the detector — ctjlewis · 2026-11-16
- Bounty hits $3,000 as researcher shows AI text detectors are trivially evadable — ctjlewis · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16