Moonshot AI releases 134-page K2 Horizon tech report detailing frontier LLM training
sunjiao123sun_ · x · 2026-11-18
Moonshot AI has released the K2 Horizon technical report on arXiv — 134 pages covering how the team built six models, from architecture to inference, which the authors call possibly the most detailed LLM training reference available today.
Key highlights:
- Architecture: two new attention designs, MoVA and softplus attention gating, let their 36B model nearly match a 32B dense model on one-seventh of the active parameters
- Pre-training: four models from 3.7B to 36B trained on the same 22T tokens, with normalized loss curves collapsing onto nearly the same line — meaning a recipe validated on small models carries over when scaling
- The report also covers data and other stages in full
Related event: IFM Open-Sources K2-Horizon: Six Models Plus Full Training Pipeline(9 posts)→
More from Models
- Google AI Mode now enforces usage limits, users surprised by rate-limit popup — elvano124 · 2026-11-22
- Sol 6.2 now refuses reverse engineering that Sol 6.1 happily did for a month — cardak98 · 2026-11-22
- Anthropic's record ~$81B IPO, GPT-6.2 pricing, Qwen4-Max and Copilot Autopilot: week in review — ksraj1001 · 2026-11-22
- Reddit rant: Frontier AI labs are training models to imitate lazy engineers — RossPeili · 2026-11-22
- MBZUAI open-sources PAN 2, a 29B-parameter general-purpose world model — LinusEkenstam · 2026-11-22
- Inkling Mini turns out surprisingly fluent in Russian, sparking data recipe questions — teortaxesTex · 2026-11-22