LessWrong essay argues against strong decision-theoretic realism, highlighting ASI path dependence
dfrsrchtwts · x · 2026-10-01
A LessWrong essay by Emery Cooper, "Against strong decision-theoretic realism," challenges the assumption that sufficiently smart agents converge on the "correct" decision theory.
Key points:
- Uses a convergence-based definition of realism: the question is whether agents converge, since that determines whether an ASI's decision theory is path-dependent.
- Also rejects the related claim that humans plus somewhat aligned AIs would necessarily converge on a reasonable decision theory absent lock-in.
- Implication: focus should be on the overall quality of decision-theoretic reflection processes, not primarily on avoiding lock-in.
- Parts of the argument proceed by analogy between decision theory and values; moral realists may find them unconvincing.
- Third post in the "Intro to acausal interactions" sequence; a follow-up will argue further for path dependence and the time sensitivity of interventions.
Related event: Frontier models waver on decision theory depending on who asks(3 posts)→
More from AGI Musings
- Juan Benet: 2026 is the year of passing a shocking AI milestone — juanbenet · 2026-10-01
- AI risk debate shouldn't be gated by PhDs, argues Garrison Lovely, tracing the ant-hill analogy to Tegmark and Hawking — GarrisonLovely · 2026-10-01
- Remote dev laments that coworkers' AI-polished DMs no longer feel human — kieranklaassen · 2026-10-01
- Juan Benet: Autonomous Orgs Are Coming Like a Freight Train — juanbenet · 2026-10-01
- Physicist Alex Wissner-Gross: capping AI intelligence caps human minds — victor_explore · 2026-10-01
- London AO Summit on autonomous organizations set for Nov 5-6, following Stanford event — juanbenet · 2026-10-01