K2-Horizon drops 134-page technical report on six fully open models up to 375B-A23B
HongyiWang10 · x · 2026-11-18
The K2 team (IFM, led by Eric Xing) released the K2-Horizon technical report (arXiv:2611.21003) — 134 pages covering six fully open models from 0.9B dense to 375B-A23B MoE, billed as possibly the most detailed LLM training reference available today.
- Architecture: two new attention designs — MoVA and softplus attention gating — let the 36B model nearly match a 32B dense model with 1/7 the active parameters
- Pre-training: four models from 3.7B to 36B trained on the same 22T tokens, with normalized loss curves collapsing onto nearly the same line, so small-model recipes carry over when scaling
- 94 authors including Zhengzhong Liu and Eric Xing
Related event: IFM Open-Sources K2-Horizon Fleet up to 375B(3 posts)→
More from Models
- Indie dev's model scans raw binaries directly, scoring 0.977 on code blocks — wavefnx · 2026-11-18
- Insider claims Gemini closed the CUA gap to 0.9% with OpenAI on Agents' Last Exam — allenainie · 2026-11-18
- OpenAI engineers secretly use Astra for Law to write PRDs and design docs — sherwinwu · 2026-11-18
- Astra 6.2 is likely an improved checkpoint, not Bel — insider speculates Bel lands as 6.5 — haider1 · 2026-11-18
- Microsoft launches seven MAI models at Ignite, matching GPT-6 quality for under 2% of the cost — marlene_zw · 2026-11-18
- Google insider: Gemini CUA now just 0.9% behind OpenAI's just-released model — allenainie · 2026-11-18