muP Lets You Tune Hyperparameters on a Small Model and Transfer to Giant Ones
RobertTLange · x · 2026-11-13
Robert Lange breaks down the Tensor Programs V paper: under standard parametrization, a learning rate tuned on a small model fails at larger widths — updates grow with width and the tuned η diverges.
μP (Maximal Update Parametrization) scales η per layer with width so update size stays constant, enabling muTransfer: tune on a small proxy model, then zero-shot transfer HPs to the full-sized model. Verified on Transformers and ResNets: HPs transferred from a 13M model beat published BERT-large (350M) numbers; transferring from 40M beats published 6.7B GPT-3 results at only 7% of total pretraining cost in tuning. Open-sourced via pip install mup.
More from Research
- Swabbing 96 sites: IIT Madras maps Chennai Metro's urban microbiome — ravi_iitm · 2026-11-13
- 38,101 ICLR 2027 Submissions Reveal an LLM Naming Crisis: 47 TRACEs, 42 PRISMs — zetalyrae · 2026-11-13
- Open-weights AI text detector ParaTrace v8 hits 99.2% TPR at 1% FPR, 5th on RAID leaderboard — AltruisticCouple3491 · 2026-11-13
- A curated 9-paper reading path into LLM quantization for beginners — East-Muffin-6472 · 2026-11-13
- HF open-sources Ettin rerankers: six sizes, SOTA, full training recipe — antoine_chaffin · 2026-11-13
- LightOn open-sources DenseOn: SOTA single and multi-vector retrieval models — antoine_chaffin · 2026-11-13