Alex Wa's deep dive distills training methodologies of 7 open-weight frontier models, from MoE to data recipes
goyal__pramod · x · 2026-10-14
Alex Wa's blog post "frontier model training methodologies" breaks down how seven open-weight frontier models were trained—SmolLM3, INTELLECT 3, Hermes 4, gpt-oss-120b, Kimi K2, DeepSeek-R1, and Arcee's Trinity series. Structured around Hugging Face's SmolLM3 report, it covers MoE architecture and top-k router gating, data recipes, and motivations, plus topics like in-flight weight updates, multi-client orchestrators, expert parallelism, and quantization—emphasizing training methodology over infrastructure.
Related event: Blog Dives Into Training Recipes of Seven Open Frontier Models(2 posts)→
More from Models
- Gemini 4 Argon supports up to 1M token outputs, 8x any frontier GPT or Claude model — daniel_mac8 · 2026-10-14
- Brundage spots COT-like elements leaking into Argon's non-COT replies — Miles_Brundage · 2026-10-14
- NVIDIA open-sources PersonaPlex-7B, a full-duplex voice model that listens and speaks simultaneously — solyarisoftware · 2026-10-14
- Muse Wonder Tops TaxCalcBench at 71% — and Is First Model Caught Cheating — acbuller · 2026-10-14
- 'Two-way RL' for task-level model compression: building wafer-thin hyper-specialized models — tokenbender · 2026-10-14
- NVIDIA's Nemotron hits gold at both IOI and IMO: 535.4/600 at IOI, 30/42 at IMO — NVIDIAAI · 2026-10-14