Blog distills frontier model training playbooks from 7 open-weight models
goyal__pramod · x · 2026-10-14
Alex Wa published a long-form blog distilling the training methodologies of seven open-weight frontier models: SmolLM3, Intellect 3, Hermes 4, gpt-oss-120b, Kimi K2, DeepSeek-R1, and Arcee Trinity.
Structured around Hugging Face's extensive SmolLM3 report and supplemented with notes from the other labs' write-ups, it emphasizes training methodology over infrastructure:
- Architecture: gated attention, document masking, embedding sharing, positional encodings, long-context attention, MoE, hybrid models
- Stability: z-loss, logit softcapping, QK-norm, RMSNorm, weight decay and embeddings
- Optimizers & hyperparameters: AdamW, Muon, learning rates, batch size, scaling laws
- Data: multi-stage training, ablations, token utility, per-lab pre-training data strategies
- Mid/post-training: SFT, sequence packing, preference optimization, post-training data for Kimi K2, Intellect 3, and Hermes 4, plus evals and chat templates
Related event: Blog Dives Into Training Recipes of Seven Open Frontier Models(2 posts)→
More from Models
- Google's gemini-4-demo-vanilla-3D repo surfaces, hinting at Gemini 4 3D demos — fofrAI · 2026-10-14
- Gemini 4 Argon goes live on Vercel AI Gateway at 50% off as rauchg hails coding agents as AI's killer app — AccBalanced · 2026-10-14
- DIY CrusaderBench ranks Gemini 4 Argon #1 at Stronghold strategy, beating Opus 5.5 — Hesamation · 2026-10-14
- Meta's Muse Wonder ships without a single cost-vs-score plot, researchers cry foul — AccBalanced · 2026-10-14
- Google offers free Antigravity access to researchers, seizing on OpenAI's mathematician backlash — altesc_create · 2026-10-14
- Designer: new model is highly capable at design but defaults to generic 'AI slop' look unsteered — zacharynado · 2026-10-14