Rationalization Theory Must Model Bounded Rationality
geoffreyirving · x · 2026-07-08
The author proposes that a "good rationalization theory" must account for bounded rationality: with infinite compute, an AI wouldn't need to guess, eliminating the heuristic errors that misaligned goals could exploit.
They emphasize that post-hoc rationalization is not a causally accurate record of the reasoning process. The concept of "expanding on demand" can be misleading because heuristics do not equate to full reasoning.
Related event: Geoffrey Irving: AI Safety Must Solve Post-Hoc Rationalization(8 posts)→
More from AGI Musings
- AI accusations shake literary prizes from Caine to Goncourt: who decides what's human? — TuhinChakr · 2026-10-26
- Tech discovers wargaming: applying to be Anthropic's tabletop exercise lead — round · 2026-10-26
- RSI is the culmination of the industrial revolution: a factory building a faster factory — tobias_rees · 2026-10-26
- NYT roundtable asks five experts what the rules for AI companies should actually be — ShakeelHashim · 2026-10-26
- Anti-AI content is algorithm gold and very profitable, quips commentator — annetgriffin · 2026-10-26
- Silicon Valley breakdown: five Stanford-area giants worth ~$18T — FinanceYF5 · 2026-10-26