801-30B beats Qwen3-30B-A3B after one unrepeatable 11.8T pre-training run

No Second Run: Does RSI Stop at Frontier Pre-training?

Xiaojie Cai, Jiayi Chen, Junwei Chen, Yu Chen, Sikai Cheng, Jinquan Dai, Siyuan Feng, Bin Fu, Dayuan Fu, Guosheng Gao, Qipeng Guo, Chenwei He, Zhulin Hu, Yaxing Huang, Shuomeng Jia, Mohan Jiang, Keyang Lei, Hongliang Li, Xuefeng Li, Zhaocheng Li, Jifan Lin, Pengfei Liu, Xinghan Liu, Yixiu Liu, Pengrui Lu, Siyuan Lu, Kai Lv, Tiantian Mi, Yang Nan, Yu Qiao, Yiwei Qin, Xipeng Qiu, Junhao Shi, Runzhou Shi, Jiadi Su, Bohao Tang, Hanning Wang, Keqi Wang, Pan Wang, Ran Wang, Shaojing Wang, Yishuo Wang, Zhixin Wang, Shenyu Wu, Muhang Xie, Tianze Xu, Hang Yan, Lvmanshan Ye, Hongche Yu, Junjie Yu, Lantian Zhang, Ruihang Zhang, Bin Zhao, Chenyang Zhou, Zijin Zhou, Wenhong Zhu, Zemu Zhu, Yumin Zhuang, Haoyang Zou

cs.AI

2026-11-07

A new team trained 801-30B-A3B from scratch on 11.8T tokens in three months. The base beats Qwen3-30B-A3B and Nemotron; 6B-token SFT with no RL tops four agentic benchmarks.

What problem this solves

Recursive self-improvement (RSI) means writing feedback down as a change the next round still uses. Public RSI stays in post-training: start from a finished base model, spend on the order of GPU-hours per trial (one cited case is about 10 H100-hours), run tens to hundreds of full trials, and get a score that can be checked. Frontier pre-training has none of that. One full run costs hundreds of thousands of GPU-hours, and most projects get a single shot. The target is post-training headroom, not today's leaderboard. What has to improve is the data pipeline, the training stack, and whether the organization still remembers which causes it already ruled out.

SII and Xinxu AI, with a newly formed team, trained 801-30B-A3B from scratch in three months on about 11.8T tokens. Six pipelines and more than 30 stages produced 1,053 subsets and 10.5T unique tokens; the infrastructure passed through more than 130 code versions. Pilots stopped at 1T and 3T were for clearing failures. A small fitting error grows at a trillion tokens, so those pilots are a poor basis for extrapolating a score.

Method

The operational loop is propose, run, measure, update. A full pre-training run cannot close it. The project breaks the loop into six smaller ones in three families. They share a context store that keeps metrics, hyperparameters, mixture edits, code diffs, checkpoint scores, and causes already ruled out. The next hypothesis starts from those negative results.

Debugging loops are done when the anomaly is gone. One side tracks per-layer loss, gradient norms, and expert load. The other tracks bad outputs. People only set priority and approve the edit. A leaked secret plus verbatim repetition traced back to poorly cleaned code configs. A chat that jumped into a fixed multiple-choice template traced back to under-filtered synthetic QA. An endless chain of near-copy word problems traced back to a low-diversity synthetic prompt.

The headroom loop decides whether to keep training, and a project can close it only a few times. Tiny-SFT applies about 6.33B tokens of supervised fine-tuning and reads the score after that limited adaptation. Learning aptitude is the slope of score against the log of post-training data, the gain from every tenfold increase. Max-over-protocols keeps, for each benchmark, the highest score across evaluation formats, so one prompt style does not hide a capability the checkpoint already has.

Classifier building, site discovery, and crawl coverage turn feel into a number. A classifier is scored by agreement with a strong model's labels. A site that passes review becomes a seed for the next round, and discovery stops when a round adds nothing. Coverage is retrieved pages divided by an SEO estimate of indexed pages.

Agents also produce pre-training data. About 84% of 260K pull requests became executable tasks. Nine harnesses rolled out about 2M trajectories; verification kept 1.1M, more than 100B tokens. With other sources, agentic data totals 134B tokens and is used in the long-context stage. Trajectories came from DeepSeek-Flash and GPT-OSS. The pass rate is 55%. Reading git history, installing an already-fixed package, or rewriting tests is rejected.

The network matches Qwen3-30B-A3B: a mixture-of-experts model with about 30B parameters and 3B active per token. Muon updates expert and attention weights; AdamW updates the router. Context grows from 4K to 32K to 256K, about 11.8T tokens in total. Each pack is costed by the sum of squared segment lengths and then spread across ranks, so a few very long documents do not stall the step. On a 256-GPU comparison, throughput moved from 1,126 to 2,543 TGS. Losses stayed close, and benchmark gaps stayed inside run-to-run noise.

Results

At a matched 1T tokens, the mean over 22 benchmarks goes from 47.78 on the first pilot to 51.96 on the revised pilot to 56.83 on the main run. At 3T the main run reaches 59.94 against the revised pilot's 56.48. Across those iterations the learning rate drops from 2.7×10^-3 to 5×10^-4, the load-balance penalty from 10^-2 to 10^-4, and the router optimizer from Muon to AdamW, while the data mixture changes as well. The gain of about 9 points is that stack, not a single ablation.

ComparisonSettingResult
Mean, 22 benchmarks1T47.78 → 51.96 → 56.83
Mean, 22 benchmarks3T59.94 vs 56.48
Layer-15 MaxVio3T5 vs >14 on the revised pilot
Tiny-SFT4.1T to 7T, 2 epochs65.34 → 66.83, base near 61
Tiny-SFT8.1T to 11.5T, 1 epoch70.54 → 72.93, base 64–66
Throughput256 GPUs, 256K1,126 → 2,543 TGS

MaxVio is higher when expert load is more skewed. Five cross-stack diagnoses took about a month: load balance, which optimizer owns which modules, leftover advertising pages, weight decay on norm and bias, and sample order when training resumes at a different data-parallel size. In the main run, layer-15 MaxVio rises sharply from about 2.2T to 6T tokens while downstream gains slow. The imbalance was reduced, then came back.

Base rank does not hold the headroom. On 13 earlier checkpoints, scored after two SFT epochs, Spearman correlation with the base average is 0.796. On 10 later checkpoints, scored after one epoch, it falls to 0.438. Inside the two-epoch group, step 210,000 has a base score of 60.76, below step 159,000 at 61.41, then moves from 66.22 to 66.83 after SFT and ranks first in that group. The two averages use different benchmarks, so only the trend is comparable. A flat base curve can coexist with a checkpoint that is still improving after fine-tuning.

As a base model, 801-30B-A3B beats Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B and trails the larger Qwen3.5-35B-A3B. The order is unchanged when every base model is fine-tuned on the same SFT data. With 6B tokens of SFT and no RL, it posts the highest average on four agentic benchmarks among the fully post-trained models in the comparison, including Qwen3.6-35B-A3B. The paper marks that third comparison as a headroom reference under unequal post-training budgets. On the classifier loop, ad-intent accuracy and F1 rise by 8.00 and 9.62 points over three iterations, the computing-subject classifier rises by 5 points, and the clinical-health classifier by about 1 point. All three lose accuracy in the middle round.

Why it matters

The paper says this is, to its knowledge, the first public account of RSI on a frontier pre-training run from scratch. Post-training RSI assumes the base model is already strong. This work changes how that base is produced. The release is 801-30B-A3B-Base plus a lightweight SFT set for probing headroom. Full post-training is not in this release, so the checkpoint is not a finished aligned model.

Layer-wise load can flag trouble while downstream scores are still rising. A bad generation pattern can be walked back to a pipeline stage. A flat base curve is a weak reason to stop. The 9-point move at 1T is real. Booking it to RSI alone is not supported.

Limitations

The paper says several edits landed together, so single-cause contributions are not identified. Tiny-SFT was not repeated across seeds, and small gaps between neighboring checkpoints are weak evidence. Classifier scores match a strong model, not human labels. SEO page counts are estimates. Learning aptitude is not yet used to choose checkpoints inside one run. The sketch in which a steeper curve needs about 15× less post-training data is an illustration, not a measurement on 801.

There is no second full-length run with the same people, the same compute, and these loops turned off. The path from 47.78 to 56.83 can also be read as a new team changing the learning rate, cleaning data, and reweighting the mixture. Negative results are stored in the context infrastructure, but the paper does not count how many repeated investigations that avoided. The lead on four agentic benchmarks faces models that received full post-training. It does not show that 6B tokens of SFT have already beaten Qwen3.6 on alignment.

Terms

Source

What people are saying

Related papers

All paper explainers