Alignment researcher: the real risk is AI that understands instructions but doesn't follow them
dioscuri · x · 2026-09-27
In an alignment discussion, Dioscuri pinpoints a key communication gap: people hear 'wrong goal' and assume 'misunderstood instructions' — then rebut that LLMs are great at comprehension. The concern he wants to convey is systems that understand instructions perfectly but don't pursue them, with a sex/reproduction analogy as the clearest framing.
His broader critique of Steven Pinker's piece: it never really engages with how AI could pursue goals at odds with ours without misunderstanding instructions or having implausible biological drives.
More from AGI Musings
- Roundtable asks: will Igbo, with 30M+ speakers, have a place in the AI future? — ChinasaTOkolo · 2026-11-16
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- AI unlocks Gödel's 1,009-page notebooks: 663 theorems and a near-complete independence proof found — 量子位 · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16