Long-read: an ASI may preserve humanity while it still needs our infrastructure
usandholt · reddit · 2026-09-21
A long-form thought experiment on existential risk, arguing we can reason about an ASI's instrumental incentives even if we can't predict its strategies.
Key points:
- Computation has physical constraints: any recursively self-improving ASI needs electricity, chips, cooling, manufacturing and maintenance — currently all provided by human civilization — so it may have reasons to preserve us, not from valuing humanity but from dependency.
- Robotics removes that dependency: an ASI that can autonomously mine, run factories, build processors and replicate robots loses a key reason to keep human civilization intact.
- The uncomfortable asymmetry: humanity likely loses effective control before the ASI stops needing us; by the time dependency disappears, we may no longer be able to intervene.
- Expanding beyond Earth may reduce conflict over terrestrial resources, but doesn't guarantee aligned objectives.
- Conclusion: compute scarcity can't predict ASI behavior but helps track shifts in instrumental incentives — cognitive superiority, loss of control, and industrial independence are three distinct milestones we tend to conflate.
More from AGI Musings
- Astro Teller: In 15 years we'll talk about hacking biology, not AI — Scobleizer · 2026-09-21
- Pensions funding the apocalypse? Open models closing the gap put OpenAI/Anthropic premium in question — AlexTensor · 2026-09-21
- sarahookr: Unpredictable pricing and IP concerns swing the pendulum toward custom AI workflows — sarahookr · 2026-09-21
- Naval: AI models trained on open web should be opened after ~12 months — rohanpaul_ai · 2026-09-21
- Jev and the bittersweet lesson: can differently sized models serve different apps? — amankhan · 2026-09-21
- Specialized Models Will Beat General LLMs: The Coming 'PC AI' Era — markjeffrey · 2026-09-21