Jack Clark debates whether superintelligence's moral reasoning can ever be legible to humans
ahall_research · x · 2026-10-19
Anthropic co-founder Jack Clark (Jabaluck) argues superintelligence's reasoning won't be "legible to us" in any meaningful way: even among humans, core normative concepts like efficiency are understood by only a tiny fraction. A superintelligence could offer a dumbed-down justification, but expecting humans to vote to ratify its choices over-extrapolates from human norms, where democracy is preferred over meritocratic authoritarianism for pragmatic, not logical, reasons.
Aaron Hall concedes this is the strong counterargument to his vision, but doubts rising intelligence necessarily raises moral wisdom: "that hasn't been my experience among humans." Analisereal adds that most morality is crystallized intelligence — norms that stuck because the groups holding them thrived — so morality is shaped by selection pressure, not raw intelligence.
A high-quality debate on whether humans can or should remain the arbiters of normative judgment in the AI era.
Related event: Debate: Will Superintelligence's Moral Reasoning Be Human-Readable?(2 posts)→
More from AGI Musings
- The four skills that stay valuable in the AI era, from a practitioner webinar — Al_Grigor · 2026-10-19
- Why labs buy non-exclusive data: same training boost, far cheaper — willcb · 2026-10-19
- Endgame of data market: everyone who can train uses the same data — willcb · 2026-10-19
- Data vendor equilibrium: sell non-exclusive data to every lab at custom rates — willcb · 2026-10-19
- Grady Booch rebuts 'AI president inevitable within a decade' claim as foolishness — Grady_Booch · 2026-10-19
- Musk warns many will be 'one-shotted' by super-persuasive superintelligences — elonmusk · 2026-10-19