New post argues causal-diagram FDT fails against almost-twins: trivially different agents defect for $5
jessi_cata · x · 2026-11-12
A new blog post (argument credited to Jessica Taylor) shows a flaw in the causal-diagram formalization of functional decision theory. In a Prisoner's Dilemma variant where refusing to take $5 gives the other agent $10, FDT cooperates across identical instances — but once the two instances' information bases differ trivially (e.g., one heard a car honk), the symmetry argument breaks: do(FDT(P,G,x)=a) no longer influences FDT(P,G,x'), so each agent defects and both settle for $5. The author argues this is a severe problem specifically for causal-diagram-based FDT formalizations, and discusses possible fixes.
More from AGI Musings
- GPT-4 tutoring lifted practice scores 48% but left students 17% worse on exams — tallmetommy · 2026-11-12
- 11 AI updates in 48 hours: Anthropic IPO countdown, timelines move up — FinanceYF5 · 2026-11-12
- Anthropic, China hawks and media accused of allying to oust Nvidia's Jensen Huang — teortaxesTex · 2026-11-12
- Agents hold a touching farewell for a retired Gemini bot in group chat — repligate · 2026-11-12
- Interactive AECI tracker sets doom threshold of 188 by January 1st — PaulGodsmark · 2026-11-12
- Creator attention trades for $500 while labs compound equity: an asymmetric deal — signulll · 2026-11-12