ICML 2025 paper: incremental AI progress alone poses systemic risk of human disempowerment
DavidSKrueger · x · 2026-11-04
David Krueger shared links pairing Addy Osmani's article "Critical thinking when agents do the work" with the ICML 2025 paper Gradual Disempowerment (Kulveit, Douglas, Ammann, Turan, Krueger, Duvenaud).
Key arguments from the paper:
- Standard AI risk scenarios depict sudden loss of control via capability jumps or coordinated betrayal; this paper argues incremental capability gains alone, with no coordinated power-seeking, pose substantial systemic risk of eventual human disempowerment.
- The core driver: machine alternatives becoming more competitive than humans across economic labor, decision-making, artistic creation, and even companionship.
- Societal systems have stayed aligned with human interests only because human participation was necessary; once displaced, institutions' growth incentives become untethered from human flourishing.
- Decision-makers who resist these pressures will be displaced by those who don't, and pressures bleeding across systems make coordination difficult.
Full version available on arXiv.
Related event: Addy Osmani warns critical thinking is the scarce skill of the agentic era(4 posts)→
More from AGI Musings
- Noam Brown's 2023 bet lands: 10K agents solve Navier–Stokes, Riemann proof claimed — haider1 · 2026-11-23
- 2004 Letter Predicted Mathematicians Would Be the First Profession AI Makes Obsolete — ericwdolan · 2026-11-23
- If AI proves Riemann, formal verification becomes the infrastructure of all software — inductionheads · 2026-11-23
- Can We Engineer Serendipity? Rethinking social networks with agents — Borthwick · 2026-11-23
- Should lab mathematicians get better models than the public? — inductionheads · 2026-11-23
- LeCun: we supposedly have superintelligence but still no level-5 autonomous driving — ylecun · 2026-11-23