Tsinghua Benchmark: Over Half of AI Agents Fail to Improve Training Algorithms
alex_verem · x · 2026-08-23
Researchers from Tsinghua University and MosaicML released AI4AI-Bench, a benchmark to test AI agents' ability to redesign training algorithms for recursive self-improvement (RSI). Spanning 10 real research repositories, agents are given 4 hours to rewrite training code, which is then retrained for up to 12 hours and scored. Results show a mean score of 0.166 (baseline 0.1, max 1.0) across 29 configurations of 6 frontier systems, with over half of the agents failing to modify the code at all.
Related event: AI4AI-Bench Shows Recursive Self-Improvement Remains Very Hard(3 posts)→
More from AGI Musings
- Economist argues model providers face hard limits on vertical integration into white-collar sectors — sethlazar · 2026-10-16
- METR outlines three blind spots in public knowledge of AI lab internal agent risks — CFGeek · 2026-10-16
- LMR-BENCH at EMNLP 2025 tests whether LLM agents can reproduce real LLM research — ZhiyuChen4 · 2026-10-16
- Deep Research becoming useful to experts will signal RSI arrival, researcher predicts — ZeeshanZiaML · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Ben Affleck calls AI 'just matrix multiplication'; Grady Booch echoes the skepticism — Grady_Booch · 2026-10-16