Redwood researcher: AI research automation and runaway recursive self-improvement can come apart
deanwball · x · 2026-11-04
A long-form thread by Redwood Research's w01fe (retweeted by deanwball) dissects the concept of "recursive self-improvement" (RSI).
Key points:
- As models automate more AI research tasks, many worry this could dramatically accelerate progress and leave humanity with less control.
- Automation and runaway acceleration are often conflated but can come apart: human research alone could uncover a new architecture or scaling method that speeds things up without models doing more of the work.
- Conversely, fully automating AI research might barely speed up progress if the bottleneck is still waiting on experiments.
- At a given pace of progress, more automation is probably net good, freeing humans for oversight and alignment.
More from AGI Musings
- One neuron tweak could make a model a Nazi genius, alignment headache ahead — chrisalbon · 2026-11-04
- Cambridge's David Krueger: A global AI pause needs coordination, not trust — DavidSKrueger · 2026-11-04
- Beff Jezos: Doom panic over AI raised the odds of an Orwellian 1984 outcome — beffjezos · 2026-11-04
- Reddit poster rebuts "AI is theft": if training is stealing, human school is organized crime — arknightstranslate · 2026-11-04
- AI doomers mocked: optimism demands evidence, doomsday just needs a long thread — VraserX · 2026-11-04
- Joseph Jacks predicts humanity will engineer and transcend its species within 5 years — JosephJacks_ · 2026-11-04