MoE Training Cost Cut: Proxy Models Predict Optimal Learning Rates at Trillion-Token Scale
burny_tech · x · 2026-08-25
Addressing the prohibitive cost of hyperparameter tuning for trillion-token Mixture-of-Experts (MoE) models, this paper presents a compute-efficient, two-step transfer framework.
- Methodology: The approach first uses Maximal Update Parameterization (μP) to transfer optimal learning rates across model widths. It then establishes a predictive scaling law to extrapolate these rates from short training runs to massive token horizons (e.g., 10T tokens).
- Validation: The method achieved high fidelity (R²=0.95) in extrapolating ideal learning rates for large-scale training using data from small proxy models. The authors successfully pretrained a 155B MoE model (17B active params) using configurations predicted from much cheaper runs.
- Impact: This validates that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs, eliminating the need for expensive sweeps at scale.
More from Research
- Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16
- Reasonable post-trains Nemotron 3.5 on 3,000 Verus proofs, lifting pass@1 ~30 points to 73.3% — fhuszar · 2026-10-16
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
- SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
- After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16