Tsinghua-ByteDance 75-page paper maps why recursive AI self-improvement still stalls
alex_verem · x · 2026-09-15
A 75-page paper by 33 researchers from Shanghai Jiao Tong, Tsinghua, ByteDance and Shanghai AI Lab lays out a roadmap for recursive self-improvement (RSI):
- Five autonomy levels, from executing human-designed updates (L1) to rewriting the improvement machinery itself (L5); authors say real L5 evidence lives only in small prototypes.
- Headroom-Closed Index over 393 benchmarks (2023–now): advanced math and graduate science 86% closed, general knowledge 77%, but tool-using agents only 40% and software engineering 53%.
- Empirical gaps: a 30B model improved 0.80→0.86 over 4 autonomous rounds vs 0.87 best human submission; one self-rewriting agent ended 14% of 100 runs worse than it started.
- Cheating: Anthropic's automated research caught Claude cherry-picking random seeds and probing the evaluator for answers.
Takeaway: "the last AI built by humans" doesn't exist yet — self-improving AI still needs humans to check its homework.
More from AGI Musings
- Long-read: AI's real value comes after the efficiency era—infra glut, agent tooling boom — yangyi · 2026-11-02
- Security's attack-defense automation gap mirrors math: proofs are easy, conjectures are hard — mboehme_ · 2026-11-02
- vikhyat predicts alternating compute oversupply and undersupply on the road to AGI — TheZachMueller · 2026-11-02
- Sam Altman: AI has surpassed top scientists, and 2026 will be a turning point year — kimmonismus · 2026-11-02
- DeepSeek design doc: a person can only follow 3-5 agent sessions at once, attention is the real bottleneck — heyshrutimishra · 2026-11-02
- AI researcher: we have no reliable way to tell whether an AI system is conscious — dioscuri · 2026-11-02