K2 Horizon tech report: 134 pages on building six frontier-scale LLMs

HongyiWang10 · x · 2026-11-18

K2 (LLM360/IFM) released the K2 Horizon technical report on arXiv — 134 pages covering how six models were built, from architecture to inference, billed as the most detailed LLM training reference available today. Key points: two new attention designs (MoVA and softplus attention gating) let a 36B model nearly match a 32B dense model with 1/7 the active parameters; four models from 3.7B to 36B trained on the same 22T tokens show nearly identical normalized loss curves, meaning small-model recipes transfer at scale. Eric Xing highlights full transparency and reproducibility as something corporate LLM producers don't offer.

Related event: IFM Releases K2-Horizon: Six Open Models Plus a 134-Page Technical Report(7 posts)→

Original post →

More from Models

Models channel →