801-30B-A3B tech report: decomposed pretraining self-improvement loops on 11.8T tokens
jiqizhixin · x · 2026-11-21
SII and Xinxu AI release 801-30B-A3B-Base with an official technical report on how to improve a pretraining run you can't afford to repeat.
- Bottleneck: frontier-scale pretraining is too expensive to rerun, making a single end-to-end self-improvement loop impractical.
- Mechanism: decompose into smaller loops — infra debugging (internal metrics → root-cause attribution), data debugging (output patterns → data source/pipeline attribution), a recipe loop (pilot runs at 11T/31T tokens before the 11.81T-token main run, measured via Tiny-SFT learning aptitude), classifier development, site discovery, and crawl-coverage loops; judge post-training headroom and turn experience into measurable criteria.
- Context infrastructure: shared cross-team context records connect evidence, preserve failed attempts and their explanations, and support changes mid-run.
- Result: a 30B-A3B MoE base model trained from scratch on 11.8 trillion tokens over three months, with evaluation of both the base and lightweight supervised models.
More from Infra
- SpaceX lenders resist $40B of AI chips in space: collateral would be unrecoverable — basedjensen · 2026-11-21
- Inkling 2 Mini hits 613 tok/s decode and 1M context on one RTX 5090 via Basalt — jesdga95 · 2026-11-21
- NVIDIA paper: frequent context compaction cuts LLM serving cost up to 7x vs long context — khademinori · 2026-11-21
- Analyst speculates OpenAI's problems stem from GPU shortage, no SpaceX access — mark_k · 2026-11-21
- Researcher builds local GPU server to run 24/7 AI agent system after Mac mini runs out of memory — nodosenlared · 2026-11-21
- Custom engine runs Inkling 2 Mini at 129.5 tok/s with 344K context on a single RTX 3090, 2.3x faster than llama.cpp — teortaxesTex · 2026-11-21