Kimi K3 paper points to hybrid attention, Kimi Linear, and no positional embeddings
peterjliu · x · 2026-07-28
Kimi K3 appears to use hybrid attention and a Kimi Linear variant
The post says the Kimi K3 paper suggests the model uses a hybrid linear / full-attention design and a proprietary Kimi Linear variant, with claims that it outperforms full attention.
The author argues this may be the first confirmed case of a frontier model adopting hybrid attention, and notes how unusual it is to see such departures from standard Transformer orthodoxy in a mainline frontier model. They also point out that the paper appears to remove positional encodings/embeddings.
The implication is that Moonshot is betting on small-scale scaling results strongly enough to ship architectural choices that other teams have struggled to justify.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16