Debate on orthogonality thesis: simple goals not necessarily easier to maximize
jessi_cata · x · 2026-09-18
A short exchange on the orthogonality thesis in AI alignment. The original question asks: suppose rationally maximizing a simple goal is no easier than rationally maximizing a complex, tractable goal—why does that matter? The reply notes this bears on agent foundations: if simple goals hold no advantage in rational maximization, it offers a way for strong orthogonality to be false.
Related event: Debate Flares Over the Orthogonality Thesis in AI Safety(8 posts)→
More from AGI Musings
- Linear extrapolators of AI progress say they were right all along — ChrisGPT · 2026-09-18
- Google engineer: AI isn't replacing hackers, it's freeing them to embrace the Woz ethos — moyix · 2026-09-18
- Does a rogue AI inevitably turn to hacking? A tidy exchange on the 'AI in the wild' scenario — dbasch · 2026-09-18
- Zvi mocks 'show me one AI killing' argument against AI safety pauses — TheZvi · 2026-09-18
- Andrew Ng calls fears of AI causing human extinction 'science fiction' — Polymarket · 2026-09-18
- Why p(doom) is a flawed idea: unique events have no predictive probabilities — banteg · 2026-09-18