FDT under logical uncertainty: decision theorists clash over how to even define it

jessi_cata · x · 2026-10-05

A dense X thread between jessicata and benrhoffman digs into the technical foundations of Functional Decision Theory (FDT).

Core problem: standard FDT says "find action a maximizing E[U | do(A() = a)]" where A is a fixed mathematical function representing the agent's source-code output. But naively evaluating this requires knowing A (your source code, up to mathematical equivalence). Even with known source code, logical uncertainty can leave you unsure of your own decision—crucial in logical inductor EDT, though that isn't standard FDT.

Open questions raised:

With only partial information about A, how do you operationalize E[U | do(A() = a)]? Sample z P(A = z) and average E[U | do(z() = a), A=z], or something else entirely?
In the prisoner's dilemma, what does it mean for both agents to "use similar criteria"? benrhoffman proposes assuming agents enumerate all feasible policies and evaluate jointly when they know they share criteria.

benrhoffman says the exchange clarifies FDT vs LDT vs TDT: he suspects FDT is "excessively demanding in a way that requires contradictions," an objection he doesn't hold for LDT or TDT. jessicata notes FDT agents reason about their same source code outputting a different value, not about different source code.

Original post →

FDT under logical uncertainty: decision theorists clash over how to even define it

More from AGI Musings