Kimi K3 paper drops positional embeddings and departs from Transformer orthodoxy
bookwormengr · x · 2026-07-28
The author says Kimi K3’s paper contains a surprising number of real innovations and departures from transformer orthodoxy, arguing that it is not just “mostly a Transformer with a few Noam mods.”
They highlight one especially bold choice: removing positional encodings/embeddings entirely, and say they’ll share more thoughts after a deeper read.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Research
- Microsoft AI explains how small-model ladders guided its 2.8T-parameter MAI-Base-2 — hboelman · 2026-10-16
- Injecting hidden-state memory cartridges into frozen Mistral-7B and Qwen2.5-7B, weights untouched — Nearby_Indication474 · 2026-10-16
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- DeepMind: four Gemini agents beat Minecraft end-to-end in 27h14m with no human help — omarsar0 · 2026-10-16
- Is AI Interpretability Close to a Breakthrough? A New Essay Has Researchers Divided — The Decoder · 2026-10-16
- MagicBeanStalk paper: recursive synthesis of hard problems boosts gpt-oss-120b by 8.6 points on 8 math benchmarks — hyunw_kim · 2026-10-16