Debate Erupts Over the Orthogonality Thesis in AI Safety

On September 18, LessWrong author jessicata and X user Algon33 held a multi-round public debate on the orthogonality thesis in AI safety, spanning its precise formulation, compute constraints, and goal feasibility. No resolution was reached, but the exchange exposed ongoing disagreement within the alignment community over how rigorously this foundational concept is stated.

Confirmed

• jessicata challenged the classic formulation of the strong orthogonality thesis and clarified a misreading: it does not claim "only one goal is possible," but makes a strong specific assertion about the relation between intelligence and goals; refuting it requires spelling out what its negation actually is.
• jessicata raised a paperclip-paradox-style argument: if rational action over time is treated as a hard optimization/constraint-satisfaction problem, it is not obvious that rationally maximizing a simple goal (like paperclips) is easier than maximizing a complex but computationally tractable goal. He further asked what follows if the most efficient AI architectures are implicated, tying the question to agent foundations.
• Algon33 challenged the claim that the thesis is "true in practice" as imprecise: compute limits do exist, but why couldn't human-level intelligence hold goals radically different from human ones, and why wouldn't the space of efficient goals be large and diverse enough?
• jessicata responded that one cannot assume unlimited intelligence—compute boundaries are themselves an intrinsic constraint on intelligence.
• Algon33 also argued in defense: even a mind of human-level intelligence could have an extremely wide, nearly arbitrary range of goals, and smarter minds are even more capable—enough for the thesis to hold in practice.
• Algon33 further distinguished concepts: differences in goal difficulty (e.g., "having a few kids" is easier to maximize than "becoming the world's richest person," and the latter is simply infeasible below sufficient IQ) are separate from the thesis's claim that intelligence and goals are independent, demanding logical clarity from the other side.

Why it matters

• The orthogonality thesis is a cornerstone of AI alignment and agent foundations; if the intuition that "simple goals are easier to rationally maximize" is weakened, traditional judgments about superintelligent goal diversity and predictability could be affected.
• The debate reveals a split between the thesis's "strong formulation" and its "practical truth," underscoring persisting disputes over conceptual precision in the alignment community that may drive a more rigorous reformulation.

2026-09-18 ~ 2026-09-18 · 8 related posts

• Is maximizing a simple goal really easier? Rethinking paperclip rationality as an optimization problem — jessi_cata · 2026-09-18
• Debate: Is Maximizing a Simple Goal Really Easier Than a Complex One? — Algon_33 · 2026-09-18
• Debate on orthogonality thesis: simple goals not necessarily easier to maximize — jessi_cata · 2026-09-18
• Goal Feasibility vs Orthogonality: Untangling a Common AI Safety Confusion — Algon_33 · 2026-09-18
• [source] Orthogonality Defense: Smart Minds Can Hold an Arbitrary Range of Goals — Algon_33 · 2026-09-18
• Orthogonality Thesis Debate: Compute Bounds vs Intelligence-Goal Independence — jessi_cata · 2026-09-18
• [source] New LessWrong Essay 'The Obliqueness Thesis' Reopens The Orthogonality Debate In AI Alignment — jessi_cata · 2026-09-18

1 near-duplicate retellings: Algon_33