First RSI in Frontier Pre-training: 801-30B Beats Qwen3-30B After 3-Month Run
burny_tech · x · 2026-11-19
The arXiv paper No Second Run reports the first application of recursive self-improvement (RSI) to an entire frontier pre-training run from scratch:
- Why pre-training resists RSI: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective (post-training headroom) can't be read from a single score; and what must improve is the whole production system — data pipelines, training infrastructure, teams.
- Method: since no end-to-end loop can close over one unrepeatable run, RSI is decomposed into smaller loops, each repairing a broken step. Five cross-stack diagnoses were resolved: load balancing (global-batch balancing), optimizer module assignment (Muon vs AdamW), data quality (filtering ads/low-quality pages), weight decay exclusions for normalization/bias params, and data-loader sample-order restoration on resume.
- Result: 801-30B-A3B, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. As a base model it outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3-35B-A3B; with lightweight SFT (6B tokens, no RL) it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3-6.5-35B-A3B.
- The base model and SFT data are open-sourced.
More from Models
- Gemini 4 Neon rolls out to AI Pro/Ultra subscribers with Gmail and Docs grounding — GeminiApp · 2026-11-19
- Google launches Gemini 4 Neon for AI Pro and Ultra subscribers across all platforms — GeminiApp · 2026-11-19
- Teardown of Antigravity's Neon: 762 tool calls, 13 sessions, near-perfect bug fixes — kimmonismus · 2026-11-19
- Gemini 4 Neon benchmark table shows wins over Opus 5.5 and GPT-6.2 Sol on most metrics — imjustnewatai · 2026-11-19
- Hands-on: Gemini 4 Neon builds 17 interactive demos in a day — kimmonismus · 2026-11-19
- Hugging Face engineer: Gemini 4 Neon is faster, more token-efficient with better-tuned thinking levels — osanseviero · 2026-11-19