DeepSeek-V3's LR schedule dissected: warmup, peak, cosine decay, two-stage final anneal
cneuralnetwork · x · 2026-11-05
A developer walked through DeepSeek-V3's pretraining learning-rate schedule (14.8T tokens) with an annotated chart:
- Warmup: gentle ramp-up to stabilize early optimization
- Peak: long hold at 2.2×10⁻⁴ until 10T tokens for broad learning
- Cosine Decay: gradual reduction to 2.2×10⁻⁵ around 14.3T
- Final Anneal: two low-LR plateaus — 333B tokens at 2.2×10⁻⁵, then the final 167B at 7.3×10⁻⁶ for fine convergence
The chart is a reconstructed schematic based on the training hyperparameters in the DeepSeek-V3 technical report.
More from Models
- OpenAI takes GPT-Live outbound telephony API to GA with SIP trunk support — gdb · 2026-11-05
- RoboColosseum: 8 models fail all 5 Unitree G1 tasks across 5,699 real-robot trials — DJiafei · 2026-11-05
- Prompt slop is the wrong paradigm: Reddit argues models should just remember you instead — infieldmitt · 2026-11-05
- One AI behavior sample, three readings: doomers, whisperers, and people who actually looked — sebkrier · 2026-11-05
- Teknium teases Nous Research: "Time to cook" — Teknium · 2026-11-05
- Verda Launches Inference Preview with Day-0 Access to Kimi K3.1 Pro — Kimi_Moonshot · 2026-11-05