Kimi K3 paper drops position embeddings and pushes beyond Transformer orthodoxy
peterjliu · x · 2026-07-28
The author says Kimi K3’s paper appears full of genuine architectural departures rather than incremental tweaks.
The standout claim in the post is that the model removes position encodings/embeddings altogether, which the author describes as a bold break from Transformer orthodoxy and evidence that the team is doing real research rather than copying existing designs.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Research
- Digital-first GLP-1 program delivers 16.1 kg average weight loss across 8,000+ women — EricTopol · 2026-10-16
- Injecting hidden-state memory cartridges into frozen Mistral-7B and Qwen2.5-7B, weights untouched — Nearby_Indication474 · 2026-10-16
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- DeepMind: four Gemini agents beat Minecraft end-to-end in 27h14m with no human help — omarsar0 · 2026-10-16
- Is AI Interpretability Close to a Breakthrough? A New Essay Has Researchers Divided — The Decoder · 2026-10-16
- MagicBeanStalk paper: recursive synthesis of hard problems boosts gpt-oss-120b by 8.6 points on 8 math benchmarks — hyunw_kim · 2026-10-16