First RSI on a Frontier Pre-training Run: Open 801-30B Beats Qwen3-30B
dair_ai · x · 2026-11-12
SII & Xin xu AI released a paper claiming to be the first to apply recursive self-improvement (RSI) to an entire frontier pre-training run from scratch.
Core idea:
- Prior RSI work targeted post-training: rerunnable loops, verifiable scores, cheap runs
- Pre-training is the opposite: one run costs hundreds of thousands of GPU-hours and can't be repeated, no single score to read — what must improve is the whole production and organization system
- So they decompose the pre-training loop into smaller closable loops: agents debug infrastructure and data, probe post-training headroom via tiny SFT runs, and turn expert judgment into metrics for data pipelines, with a shared context store preserving negative results across teams
Result:
- Open 30B-A3B MoE base 801-30B-A3B (11.8T tokens, 3 months) outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B as a base, trailing only the larger Qwen3.5-35B-A3B
- With only 6B tokens of SFT and no RL, it achieves top open-model scores on 4 agentic benchmarks, including beating Qwen3.5-35B-A3B
- Base model and SFT data open-sourced at github.com/GAIR-NLP/801-30B
More from Infra
- Qwen3.8-Flash Hints at Stable Local Agents on Small Hardware — Express_Quail_1493 · 2026-11-13
- Nebius Q3 dissected: GPU-hour pricing up 20%+, over 90% of contract value prepaid — tengyanAI · 2026-11-13
- Five IndiaAI Mission-empanelled cloud providers lack data centres, barred from compute tender — kingslyj · 2026-11-12
- Startup turns Apple silicon into elastic cloud compute for AI agents, touting 512GB M5 Ultra clusters — awnihannun · 2026-11-12
- NVIDIA takes $4.5 billion H20 charge after new export licensing requirements — heyshrutimishra · 2026-11-12
- LangChain releases fireside chat with Modal CEO on sandboxes and inference at Interrupt NYC — LangChain · 2026-11-12