Kimi K3’s open weights push a nearly 3T-parameter MoE into the open-source era

机器之心 · wechat · 2026-07-29

Machine Heart’s article uses the open-sourced Kimi K3 as a starting point to trace how transformer architectures evolved from GPT-2 to today’s giant MoE models.

Key points:

Kimi K3 is described as a nearly 3-trillion-parameter MoE model with a 1-million-token context window, and as the first open-source model at roughly this scale.
The author compares it with GPT-2’s 124M parameters, highlighting a 22,580× jump in scale over seven years.
The article then walks through the technical path: vanilla decoder-only Transformers, KV cache, flash attention limitations in early implementations, linear attention as a memory-compute tradeoff, and fast-weight / DeltaNet-style updates that try to preserve useful associations without unbounded cache growth.
It emphasizes that each approach trades off capacity, recoverability, and efficiency, rather than being a simple “bigger is better” story.

Original post →

Kimi K3’s open weights push a nearly 3T-parameter MoE into the open-source era

More from Models