Pre-training ladder trick: test every recipe change on 300M–6B models first
teortaxesTex · x · 2026-11-07
Cited pre-training work details a rigorous model-ladder methodology: every recipe change is validated on a ladder of small models (300M to 6B active params) before the final run — but the ladder only helps if small models face the same conditions as the big one.
A key pitfall is data: a ladder model trained on 238B tokens (1% of the final run) sampled from the full 13.7T pool always sees fresh data (pool 58x its budget), unlike the final model. The fix is data-limited subsampling — shrinking each source's pool proportionally per ladder size to simulate data-constrained conditions.
Commentator teortaxesTex notes Chinese labs rarely publish such scaling ablations; existing papers focus on MoE granularity, while reports from MAI, TM and others go deeper into strategy.
More from Models
- Prominent user vents over OpenAI's repeated GPT 6.2 delays: 'What the heck is going on?' — mark_k · 2026-11-07
- Only 7 AI products rank top on web, mobile and revenue — Perplexity is one, a16z Top 50 shows — ChrisUniverse · 2026-11-07
- Arena weekly: Kimi K3.1 Pro tops open models, Gemini leads video, GPT Image 2x faster — arena · 2026-11-07
- MiniMax and Fireworks to host SF fireside on building M3.1 Flash for long-running agents — MiniMax_AI · 2026-11-07
- 'Hidden GPT-6.2 Astra' claim debunked: fake reveal tricked via one-line console object — teortaxesTex · 2026-11-07
- Inkling 2 Small tops open-weight models on ARC-AGI-2 at $0.27/task, weights coming — LiTianleli · 2026-11-07