MoE Training Cost Cut: Proxy Models Predict Optimal Learning Rates at Trillion-Token Scale
burny_tech · x · 2026-08-25
Addressing the prohibitive cost of hyperparameter tuning for trillion-token Mixture-of-Experts (MoE) models, this paper presents a compute-efficient, two-step transfer framework.
- Methodology: The approach first uses Maximal Update Parameterization (μP) to transfer optimal learning rates across model widths. It then establishes a predictive scaling law to extrapolate these rates from short training runs to massive token horizons (e.g., 10T tokens).
- Validation: The method achieved high fidelity (R²=0.95) in extrapolating ideal learning rates for large-scale training using data from small proxy models. The authors successfully pretrained a 155B MoE model (17B active params) using configurations predicted from much cheaper runs.
- Impact: This validates that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs, eliminating the need for expensive sweeps at scale.
More from Research
- Interpretability paper incoming: in-depth look at testing explanations on modern LLMs — burny_tech · 2026-10-16
- Better-learning models tend to be more interpretable, argues Nick Cammarata — burny_tech · 2026-10-16
- Cornell and Amazon revisit the 16x pretraining compute gap: diffusion LMs need only 3-8x — LucaAmb · 2026-10-16
- AI trained on the fraction of human intelligence that fit the page — r0ck3t23 · 2026-10-16
- Quotient Recurrent Reasoners hit 97.9% on Sudoku-Extreme with 78% fewer loops — burny_tech · 2026-10-16
- Mathematicians publish explainer for the Egyptian fraction proof in OpenAI's proofs collection — burny_tech · 2026-10-16