Dev trains a 21B MoE (3.36B active) from scratch on 70B tokens and logs the emergence
Prestigious-Taste-63 · reddit · 2026-10-28
An independent developer released Apex-3, a 20.99B-parameter MoE (3.36B active per token) trained entirely from scratch on 70B tokens — SuperBPE tokenization makes it roughly equivalent to 100B — using 64 routed experts plus 1 shared expert (top-4), Muon and DiLoCo on GH200s. A stable checkpoint is public for anyone to continue training.
Key observations from the training logs:
- The only clean phase transition is an ICL/induction-head jump: the score grew 5x within 150 steps around 1.3B–1.5B tokens (about 0.1% of the original 1.1T schedule).
- Arithmetic "emergence" is a metric artifact: correct-token probability rises continuously (add3: 0.003 → 0.224 → 0.60 after decay); exact match jumps only when probability crosses the decision threshold.
- LR decay produced a surprisingly large lift: overall exact match went from 0.52 to 0.67.
- MoE load balancing self-converged: max expert load fell from 12.8x the average to 1.07–1.38x by the end.
- Domain validation CE dropped smoothly with no grokking signal; final benchmarks: HumanEval+ 19.5%, MBPP 38.9%, GSM8K 8-shot 20%.
Caveats included: 64 problems per arithmetic task (standard error up to ±6 points), teacher-forced scoring, and no measurements before 8.15B tokens.
More from Research
- Real human laughter remains AGI-hard, argue AI improv researchers — korymath · 2026-10-28
- Johns Hopkins lands up to $52M from ARPA-H to build AI that decodes protein misfolding — JosephJacks_ · 2026-10-28
- Microsoft's Diagnostic AI published in Science and open-sourced, retested on Gemini 4 Argon — vivnat · 2026-10-28
- Machenta robot reacts and recovers in real time, trained 100% in simulation — Ed__Johns · 2026-10-28
- DeFM Depth-First Foundation Model: Why Anchor to Depth, Not Estimator Features — chris_j_paxton · 2026-10-28
- AI-powered formal verification finds real Redis/Valkey bugs, incl. a 23-char freeze — LingmingZhang · 2026-10-28