Kimi K3 report adds NoPE, full-rank gating and FP32 attention training
suchenzang · x · 2026-07-28
A thread on the Kimi K3 technical report highlights several architecture and training changes:
- NoPE remains in MLA, with the author joking that position information appears to be globally useless.
- A new sigmoided output gate is added, apparently to better propagate signals and avoid instability.
- Attention outputs are kept in FP32 during training to avoid the rounding issues seen in flash attention.
- To offset the extra memory cost, the training kernel was redesigned so the FP32 work overlaps with KV staging buffers, improving throughput.
- The follow-up note says Kimi K3 also moved the KDA output gate from low-rank to full-rank, and that this change does not actually solve the attention-sink issue it was meant to address.
More from Models
- Kimi K3 lands on Fireworks AI for inference and training — omarsar0 · 2026-07-28
- Gemini video generation adds words and blocks some harmless prompts — Individual-Cookie615 · 2026-07-28
- Polymarket now prices a 42% chance of a new Claude Sonnet by next month — Polymarket · 2026-07-28
- Kimi K3 paper details SiTU-GLU and quantile balancing for 896-expert MoE — KyeGomezB · 2026-07-28
- vLLM Collaborates with DigitalOcean to Host Kimi K3 Model — vllm_project · 2026-07-28
- Kimi K3 lands on ChatLLM with U.S. hosting and an open-source fine-tune — bindureddy · 2026-07-28