Kimi K3 scaling study says cosine decay beats WSD at each method’s best settings
stochasticchasm · x · 2026-07-28
Kimi K3 scaling study favors cosine decay over WSD
This post points to the paper’s scaling-law section. The authors say the architectural, data, and training changes together deliver about a 2.5× overall scaling-efficiency gain over Kimi K2.
A few concrete takeaways:
- They retuned key hyperparameters such as batch size, learning rate, tokens-per-parameter ratio (TPP), and model shape.
- In their comparison, cosine decay consistently outperformed Warmup Stable Decay (WSD) under each schedule’s own optimal hyperparameters.
- The authors note that WSD may have seemed competitive in prior work, but here the two schedules favor different optimal regimes, so a fair comparison requires independent searches.
- The plot shown in the post highlights that cosine decay reaches lower final loss than WSD under the best settings for each schedule.
Related event: Inside Kimi K3: Tri-axis Architecture and Hybrid Attention(34 posts)→
More from Research
- Different coding agents can produce very different estimates from the same model spec — JessicaHullman · 2026-07-28
- Progressive context extension may suit linear-attention hybrids better than short-SWA models — stochasticchasm · 2026-07-28
- Google DeepMind says LLMs can prove theorems but still cannot make scientific “jumps” — kylekabasares · 2026-07-28
- CMU’s O-VAD detects industrial video anomalies by tracking object state over time — CarnegieMellonU · 2026-07-28
- Latent Action Model talk will show how to learn Super Mario from observation alone — ceciletamura · 2026-07-28
- GLM 5 follows up with per-head Muon to balance attention-head updates — stochasticchasm · 2026-07-28