GAIR brings RSI to pre-training, open-sources 30B MoE base trained in 3 months
pmttyji · reddit · 2026-11-15
GAIR-NLP's paper 'No Second Run: Does RSI Stop at Frontier Pre-training?' claims the first application of recursive self-improvement to an entire frontier-scale pre-training run, with open release.
- Core argument: public RSI work targets only post-training (cheap, repeatable, verifiable scores), while pre-training burns hundreds of thousands of GPU-hours in an unrepeatable run with no single readable objective. They decompose RSI into smaller closable loops, each repairing a step the full run breaks.
- Result 801-30B-A3B-Base: a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. The base outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B; ranking holds under identical SFT. Benchmark charts show it leads SWE-bench Pro, KernelBench, AA-LCR, and AA-OmniScience averages after lightweight SFT against fully post-trained models including Qwen3.6-35B-A3B.
- Open source: base model coming soon; 6B-token SFT dataset (no RL) released.
Paper: arXiv 2611.10719; code: github.com/GAIR-NLP/801-30B.
More from Models
- ChatGPT outage reported as users note the service is down — k7agar · 2026-11-15
- strata update makes Qwen3.8-Flash-Next slower and dumber, v0.1.42.0 is the last good version — Lime-Lemon-7582 · 2026-11-15
- Gemini 4 Carbon Finishes ~16 Coding Tickets in 4.5 Hours, but Fable 5.2 Raises the Bar — every · 2026-11-15
- Bigger models hallucinate more, and 'hallucination is a feature,' says AI researcher — iruletheworldmo · 2026-11-15
- The recipe to solve any perception task: a few hundred examples and a fine-tuned VLM — giffmana · 2026-11-15
- AI agents go from 2.5% to 41.3% on real freelance benchmark in 13 months — kimmonismus · 2026-11-15