LessWrong: clashing intuitions on alignment plausibility, told via two causal models
davidmanheim · x · 2026-11-06
David Manheim posted on LessWrong to unpack the root of the divide between Yudkowskian doomers and 'AI-whisperers' like j⧉nus on alignment prospects.
- Pre-GenAI rationalist assumption (diagram A): reducing human values to an RL input means RL inevitably Goodharts away from true goals — alignment fails in the limit
- Foundation models changed that (diagram B): capabilities emerged from human representation, conceptually driving the 2020s schism between labs and outside safety concerns; optimists believed alignment-relevant representations could be preserved
- The missing piece is Yudkowsky's 'best AI news of 2025' — whether values and capabilities are truly entangled with a central preference vector — which determines how much to update from today's jagged capabilities
- Whether the criticism lands depends on specific future capability trajectories and how RL gets used
Framed as a brief explanation of an unsettled debate rather than novel theory, with ChatGPT-generated causal diagrams; featured on LessWrong's AI Frontpage.
More from AGI Musings
- Dev essay: why a runaway intelligence explosion isn't the risk — 'early onset decoherence' is — round · 2026-11-06
- Jeff Ladish on AI hacking, self-replication and automated AI R&D risk — JeffLadish · 2026-11-06
- Why AI writing feels annoying despite models winning literary competitions — cis_female · 2026-11-06
- Why AI writing annoys us: models can win literary prizes, the problem is style fatigue — cis_female · 2026-11-06
- METR estimates AI is already accelerating AI development by roughly 2x — harris_edouard · 2026-11-06
- Lightcone Commons Opens AI-Safety Grant Applications, Backed by Jaan Tallinn and Moskovitz — NathanpmYoung · 2026-11-06