Training run shows no instabilities and 1M context extended over ~10T tokens
stochasticchasm · x · 2026-09-11
- No training instabilities observed so far, suggesting the issue from v4 training has been resolved — a good sign for the architecture.
- Notably, the team continues with the WSD learning rate schedule and extends context length over a 10T token training run.
- The author finds it remarkable that training is sustained at 1M token context for that long.
Related event: DeepSeek V4 training details: 10T+ tokens, 1M context, stable RL(3 posts)→
More from Infra
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- AI's unusual bet: buying compute for years, selling intelligence by the token — vaibhavbetter · 2026-10-28
- OpenAI's Will Depue says compute locks in US primacy as 'end of human history' nears — willdepue · 2026-10-28
- Malaysia has overtaken Singapore in data center capacity, planned buildout 10x — bytebot · 2026-10-28
- Why Cloudflare Is the Best Platform to Build AI Agent Operating Systems — ritakozlov · 2026-10-28
- Shared KV memory makes Looped Transformers better and 4.7x cheaper: Cornell/IBM paper — yoavartzi · 2026-10-28