Pre-training trend watch: dropping LR decay for constant LR plus checkpoint averaging
rasbt · x · 2026-11-18
Sebastian Raschka (rasbt) flags an emerging pre-training recipe: no LR decay phase at all — instead a constant learning rate with averaging over the last N checkpoints (or a running weight average), as seen in Kolibri-1 and DatologyAI's models. Practically, this avoids committing to a total training length upfront.
Based on labs that share training details, he sorts recipes into:
- Classic cosine: Llama 3, OLMo 2/3, GLM-4.5, GLM-5, Kimi K3, Phi-4
- WSD (Warmup-Stable-Decay): DeepSeek V3/V4/V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano/Super, SmolLM3, MiniMax M2
- No decay (constant LR + checkpoint averaging): Kolibri-1, DatologyAI's models
The quoted DatologyAI post adds that they merged pre-context-extension models into post-extension checkpoints (with ablation gains) and used merging as a stand-in for MOPD in their RLVR recipe.
More from Models
- Every's vibe check: Gemini 4 Carbon codes better but judgment lags Fable 5.2 — every · 2026-11-18
- 'My favorite model': developers rave about Gemini 4 Carbon as Google re-enters the race — zacharynado · 2026-11-18
- Mistral's Le Chonk crowned best model name, free Vibe tier for Europe praised — TheTuringPost · 2026-11-17
- Dev exhausted 128 Claude accounts to dodge usage limits, takes week off — thomasahle · 2026-11-17
- Humans& Launches 744B-Param Collaborator Model Ampersand on Vercel AI Gateway — ericzelikman · 2026-11-17
- Claude Opus 5.5 Flagged Iranian School as Military Target in 22 of 32 Tests, Eval Finds — Polymarket · 2026-11-17