Kimi K3 claims 2.5× scaling efficiency gains from KDA savings
nrehiew_ · x · 2026-07-29
The thread continues the Kimi K3 training discussion with more detail on RL and distillation.
- The author says the RL FLOPs curves are jagged and speculates that each expert may not get that many RL FLOPs.
- That could imply that much of the compute is spent on MOPD instead.
- On pretraining, the post says there is little information about the token count.
- The report claims a 2.5× scaling improvement, largely attributed to KDA FLOPs savings.
- The authors also say they prefer cosine over WSD after hyperparameter search.
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16