First RSI applied to full pre-training: 801-30B-A3B trained from scratch in 3 months on 11.8T tokens
Xudong07452910 · x · 2026-11-13
SII-GAIR and Xinxu AI's paper "No Second Run" claims the world's first application of recursive self-improvement (RSI) to an entire from-scratch pre-training run, producing 801-30B-A3B-Base, a 30B-A3B MoE model.
- Problem: pre-training consumes hundreds of thousands of GPU-hours over months and can't be repeated, so the standard RSI loop (cheap reruns, verifiable scores) breaks.
- Approach: decompose pre-training into smaller verifiable loops — agents locate training faults, evaluate data and strategies, and share metrics/code changes/failure experience via a Context Infrastructure so knowledge accumulates across the run.
- Results: a newly formed team pre-trained on 11.8T tokens in 3 months; at the same 1T-token budget, average score across 22 benchmarks rose from 47.78 to 56.83 (joint improvements, not individually attributable). As a base model it outperforms Qwen3-30B-A3B and Nemotron-3.5-Lighting-30B-A3B; after lightweight SFT it tops four agentic benchmarks against fully post-trained models.
- Takeaway: when runs are too expensive to repeat, locally verifiable experiments plus accumulated, shared context may be the key to sustained self-improvement. Model and SFT data will be open-sourced.
More from coding & agent
- User builds a silent "Chief of Staff" Grok agent that only pings you for decisions — RachelVT42 · 2026-11-22
- Yengi: Open-source Windows AI IDE that talks live to Blender & Unity and auto-fixes its code — mdaiWorks · 2026-11-22
- Gemini 4 Neon tops Bug Hunt Bench; Antigravity vs AFK Pilot vs T3 Code compared — PawelHuryn · 2026-11-22
- Fake USDC bounty honeypots and wallet-guard lures target MCP and SKILL.md coding agents — Upbeat-Bus2143 · 2026-11-22
- Informal verification: using System One models as proof checkers to build epistemic trust in agents — LauraRuis · 2026-11-22
- NotebookLM + Google Antigravity: 4 free prompts to compress hours of research into minutes — aftahi_ai · 2026-11-22