Tensor Programs IV revisited: muP enables feature learning in infinite-width networks
burny_tech · x · 2026-11-10
Sharing the foundational muP paper (Greg Yang, Edward J. Hu, ICML 2021) alongside a discussion of whether theory can precede empirics. The paper shows standard and NTK parametrizations admit no feature learning in the infinite-width limit, proposes simple parametrization changes that do, derives explicit limit formulas via the Tensor Programs technique, and demonstrates on Word2Vec and MAML/Omniglot that these limits beat NTK baselines and finite-width networks — the basis for later hyperparameter transfer techniques.
Related event: From muP to Probabilistic Oracles: Theory-First AI Design Revisited(2 posts)→
More from AGI Musings
- Does knowing a text is AI-written make it mean less? A look at effort and authenticity — MinaLee__ · 2026-11-10
- ML research rule of thumb: assume ten groups are already working on your idea — A_K_Nain · 2026-11-10
- OpenAI staffer says academia unprepared as inference compute decides which fields accelerate — tw_killian · 2026-11-10
- If one mind is enhanced a thousandfold, are we still one civilization? — VraserX · 2026-11-10
- Beff Jezos vows to fight for 'freedom of thought and the right to bear compute' — beffjezos · 2026-11-10
- Andy Masley: Tenuous AI 'common wisdom' on capabilities and governance springs up overnight — AndyMasley · 2026-11-10