Reply to Bengio: the real explosion is throughput, not intelligence, widening the audit gap
AryHHAry · x · 2026-09-29
Responding to Yoshua Bengio's paper on AI automating its own R&D and the risk of an uncontrolled "intelligence explosion," the author argues the measurable surge today is throughput, not intelligence: most lab code is machine-written, lightly supervised R&D share is climbing fast, compute efficiency doubles, and per-milestone costs fall. That's a genuine positive loop — but a positive loop on the search path isn't automatically one on the understanding path. Automating experiments is easier than automating causal audits; if improvement explodes while interpretability stays linear, the gap between what systems can do and what we can trace widens. Once AI R&D writes its own successors, "percent automated" becomes a surface metric — the real question is who still holds the map of the mechanism when it's replaced faster than we can read it.
More from AGI Musings
- Affine 1, trained by anonymous Bittensor miners, matches Opus 4.1 on intelligence score — markjeffrey · 2026-09-29
- Pedro Domingos: OpenAI and Anthropic will soon be fat and slow like Google and Microsoft — pmddomingos · 2026-09-29
- Domingos: Real neolabs like SSI chase AGI their way; fakes like World Labs are acquihires — pmddomingos · 2026-09-29
- Quintin Pope: backdoor-style training setups are a poor stand-in for hypothesized inner optimizers — QuintinPope5 · 2026-09-29
- AI songs are now how memetic battles get fought, dev observes — jam3scampbell · 2026-09-29
- Two recursive mechanisms for an intelligence explosion: automated AI R&D expands the effective workforce — jeffclune · 2026-09-29