Hinton, Bengio and 20+ experts warn AI-run AI R&D could trigger an intelligence explosion
rohanpaul_ai · x · 2026-10-07
A policy paper on arXiv, led by Alan Chan and signed by 22 experts including Geoffrey Hinton, Yoshua Bengio, Jeff Clune, and Jack Clark, assesses what happens if AI starts automating AI R&D itself.
Key claims
- AI has jumped from seconds-long tasks in 2023 to tasks taking human experts days; most code inside frontier AI labs is already written by AI.
- The authors argue AI systems are on track to automate most — possibly all — AI R&D within a few years. If runtime costs stay comparable, the compute one frontier developer has today could run an AI workforce of millions.
- This could trigger an "intelligence explosion": years of progress compressed into months. Benefits could arrive earlier, but risks are extreme — capabilities outpacing society, loss of control over superhuman AI, and erosion of checks on power between states, companies, and government branches.
Policy recommendations: governments should urgently gain visibility into how much AI research is already done by AI (e.g., track the share of AI-run research), develop ways to steer and constrain such an explosion, and prepare responses in advance. The authors acknowledge large uncertainty but argue the stakes justify serious attention.
Related event: Hinton, Bengio and 22 Experts Warn on AI Automating AI Research(3 posts)→
More from AGI Musings
- Bindu Reddy Predicts a Robotics Breakthrough Within 12 Months — bindureddy · 2026-10-07
- AI safety researcher David Krueger calls for immediate international moratorium on frontier AI — DavidSKrueger · 2026-10-07
- Reddit Asks: After 'Drop Coding' Memes, What Job Is Next in 2026? — JUST_A_HUMAN_CX123 · 2026-10-07
- Gary Marcus on whether frontier LLMs can solve open math problems without symbolic harnesses — GaryMarcus · 2026-10-07
- Gary Marcus on the endless AI hype cycle: gloating, then disappointment within days — GaryMarcus · 2026-10-07
- Emily Bender: AI existential risk is 'fake,' doomsday talk hides real harms — AlexTensor · 2026-10-07