Rationalization Theory Must Model Bounded Rationality
geoffreyirving · x · 2026-07-08
The author proposes that a "good rationalization theory" must account for bounded rationality: with infinite compute, an AI wouldn't need to guess, eliminating the heuristic errors that misaligned goals could exploit.
They emphasize that post-hoc rationalization is not a causally accurate record of the reasoning process. The concept of "expanding on demand" can be misleading because heuristics do not equate to full reasoning.
Related event: Geoffrey Irving: AI Safety Must Solve Post-Hoc Rationalization(8 posts)→
More from AGI Musings
- The Evolution of LLM Business Models: Selling Outcomes Over Tokens — yacineMTB · 2026-07-22
- Bindu Reddy says GPT-6 is coming soon, with Alibaba, DeepSeek and Kimi close behind — bindureddy · 2026-07-22
- Bindu Reddy says the industry still lacks a way to train 20T models and scale post-training RL — bindureddy · 2026-07-22
- Advanced AI Models Are Becoming Impossible to Plug and Play — emollick · 2026-07-22
- AI suggested a better composition, and that made one user uneasy — Sydde · 2026-07-22
- The Thimble and the Waterfall: AI's Data Bottleneck and Feedback Loops — dyamins · 2026-07-22