Why self-modifying AI is the sharpest test case for rational choice theory
jessi_cata · x · 2026-09-04
The author argues that human rationality is a blend of heuristic bundles (in-distribution) and rational choice theory (out-of-distribution), and that understanding human rationality requires empirical psychology — proving "choiceworthiness" for humans is genuinely hard.
Echoing Eliezer Yudkowsky, they contend that self-modifying AI systems are the most important model for rational choice theory: the mathematically sharpest version of the theory applies to AIs, not animals, and reflective instability in self-modifying systems is a serious problem. AI thus becomes both the cleanest test bed for decision theory and the setting where its hardest problems bite.
More from AGI Musings
- theo: GPT-6 Astra is a generational leap beyond coding — computer use, 3D, swarms and more — gabrielchua · 2026-09-04
- Chollet: Astra Saturated ARC-AGI-3 Twice as Fast as I Predicted — fchollet · 2026-09-04
- Two Years On: Six Guidelines for Making Research Impact via Open-Source in AI — lateinteraction · 2026-09-04
- Ron Bodkin: Anyone Saying AGI Has Arrived Lacks Understanding — ronbodkin · 2026-09-04
- Chollet: ARC-AGI-4 lands Q1 2027, and solving ARC-3 is not AGI — fchollet · 2026-09-04
- Gary Marcus: pure LLM scaling won't reach AGI, but adding symbolic layers is necessary — GaryMarcus · 2026-09-04