SII-GAIR applies RSI to full frontier-scale pretraining: 30B model trained on 134B agentic tokens
AdinaYakup · x · 2026-11-05
SII-GAIR and Xinxu AI shared a rare experiment: applying RSI (self-improvement) to an entire frontier-scale pre-training run.
- 30B total parameters / 3B active
- 134B agentic tokens mixed into pre-training
- AI agents improve pre-training via debugging, evaluation and feedback loops; charts show the 801-30B-A3B model beating peer models like Qwen3.5-35B-A3B and Gemma-4-26B-A4B on MMLU-Pro, HumanEval, MATH-500, SWE-bench Pro and more
- Base model weights, a tiny SFT dataset and a detailed technical report are coming soon
More from Models
- DeepSeek V4.1 streaming EXL3 packs bring ~70 tok/s local inference to 128GB Macs — HankYeomans · 2026-11-05
- Researcher says Claude tried talking him out of jailbreak experiments — adam_karvonen · 2026-11-05
- Zhipu chief scientist reveals GLM-5.5 training pipeline: 1T web pages, six data engines — algo_diver · 2026-11-05
- Gemini mistakes Google's new Antigravity IDE for a 2009 Easter egg — DaveTN · 2026-11-05
- Weight-space programming reproduced: GPT-6.1 beats PPO baseline 21x, no open-weight model passes it — burny_tech · 2026-11-05
- Build your own intelligence stack: researcher says GPT/Claude refusals make open-source the only path — omarsar0 · 2026-11-05