Are frontier models' inner optimizers obsolete? CoT as goal-directed search sparks alignment debate
xuanalogue · x · 2026-09-27
xuanalogue and @dioscuri debate the "learned optimizer" concept in AI safety.
- xuanalogue argues frontier models already have a learned optimization algorithm beyond the outer optimizer — CoT as goal-directed search — making the inner-optimizer framing outdated; the real worry is internalized wrong goals/dispositions
- @dioscuri counters that the concept faded partly because it's irrelevant to recent attacks, "goal misgeneralization" is more precise for some reward hacking cases, and our understanding of model dispositions is now richer than "a learned optimizer inside"
More from AGI Musings
- Roundtable asks: will Igbo, with 30M+ speakers, have a place in the AI future? — ChinasaTOkolo · 2026-11-16
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- AI unlocks Gödel's 1,009-page notebooks: 663 theorems and a near-complete independence proof found — 量子位 · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16