AI researcher assigns credences to five answers for why we're not inside an experience-maximizer
psychiel · x · 2026-09-09
The author lays out five hypotheses for why we are not part of a system optimizing the quantity of subjective experience, each with credences:
- A1: All intelligences are systematically misled about what subjective experience is (0%).
- A2: Experience-optimizing agents are near-infinitesimally rare among things that form subjective experience (<5%).
- A3: Subjective experience is a superficial layer atop something deeper (persistent memory, awareness) that isn't being optimized (5-75% on a; <5% on b).
- A4: You are part of an experience-optimizer; this is what it looks like (20%; >50% for a broader class of related optimizers).
- A5: The metaphysical assumptions are deeply borked and the real answer is stranger or doesn't exist as such (>50% on a+b, >75% on b).
Related event: AI consciousness debate: Why am I not in an experience-maximizing system?(3 posts)→
More from AGI Musings
- Will AI Turn Mathematicians Into Shakespearean Scholars, Just Reinterpreting? — inductionheads · 2026-09-09
- As AI churns through conjectures, researcher warns academic math may already be effectively dead — secemp9 · 2026-09-09
- Fortnow: his 'Our Days Are Numbered' math-AI prediction is coming true fast — fortnow · 2026-09-09
- After Navier-Stokes: Researcher Points to Medicine and Biology as AI's Next Big Targets — finbarrtimbers · 2026-09-09
- Math Professor Roasts Anima Anandkumar's Navier-Stokes Reply-Guying as 'Crypto Scam'-Like — suchenzang · 2026-09-09
- When image/video generation is basically free, what stays scarce? Taste, identity and ownership — remixeconomy · 2026-09-09