MLST podcast argues recursive self-improvement hits diminishing returns, dismissing exponential AI X-risk fears
emax · x · 2026-09-27
emax shared the latest Machine Learning Street Talk podcast (43 min, guest: Weco co-founder Zhengyao Jiang), commenting that fears of exponential recursive self-improvement (RSI) X-risk are fearmongering, as emerging research suggests RSI will hit recursively diminishing returns.
Key podcast topics:
- Weco ran an AI coding agent for eight days rewriting another agent's harness (code, prompts, tools) with the underlying model frozen: the AIDE 85 experiment.
- What the reported gains over two years of human engineering actually demonstrate, held-out evaluation, and separating useful discoveries from reward hacking.
- Jiang's four levels of recursive self-improvement, compared with AlphaEvolve and the Darwin Gödel Machine.
- The limits: the experiment did not establish the system became a better improver.
- Closing on open-ended search, human-designed primitives, and Parameter Golf: where the next useful idea comes from when agents search inside human-designed spaces.
Related event: Weco's eight-day AI self-improvement experiment fuels RSI debate(4 posts)→
More from AGI Musings
- David Khourshid: AI motion design wave proves human taste is the moat — DavidKPiano · 2026-09-27
- The rarest human skill in the AI era: subtracting instead of adding — jonathan_wilke · 2026-09-27
- Diamandis talks agentic economy with Emad Mostaque and Cathie Wood: energy, datacenters, stablecoins, robotics — PeterDiamandis · 2026-09-27
- Why transformative AI could push interest rates up, not down — and crowd out everything else — banaca4 · 2026-09-27
- Analysis of 769 task logs: AI agents propose 55% of methods, humans make 85% of calls — The Decoder · 2026-09-27
- Australian universities embrace AI marking; academics warn of higher-ed collapse — nordicinst · 2026-09-27