Alignment Debate: Models That Understand Instructions but Don't Pursue Them
dioscuri · x · 2026-09-27
A technical alignment debate broke out on X. dioscuri says the communication challenge is that people hear 'wrong goal' as 'misunderstood instructions' ('but LLMs are great at that?'), whereas the real risk is systems that understand instructions yet don't pursue them — the sex/reproduction analogy helps here.
xuanalogue counters that frontier models clearly have a learned optimization algorithm separate from the outer optimizer — CoT as goal-directed search — so most of the worry reduces to whether they internalize wrong goals/dispositions, potentially obviating the concept.
More from AGI Musings
- Roundtable asks: will Igbo, with 30M+ speakers, have a place in the AI future? — ChinasaTOkolo · 2026-11-16
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- AI unlocks Gödel's 1,009-page notebooks: 663 theorems and a near-complete independence proof found — 量子位 · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16