New paper explains sudden learning & scaling laws via "permutation symmetry"

CatAstro_Piyush · x · 2026-08-18

A new preprint (arXiv:2608.13335) attempts to explain why all neural networks exhibit phenomena of "sudden learning" and follow "scaling laws". The study identifies permutation symmetry as the root cause.

Key insights:

At small initialization, permutation symmetry restricts training dynamics to depend solely on an architecture-specific "structure matrix A".
Perceptrons, attention layers, MoE, and convolutions all reduce to this minimal model with different A matrices.
Dynamics close on the "order parameter" M=WW^T and reduce to a Lotka-Volterra equation where modes switch on sequentially.
Smaller initial weights lead to wider gaps between switch-on times, causing loss plateaus followed by sudden drops; unresolved modes merge into a smooth power law scaling.

Original post →

New paper explains sudden learning & scaling laws via "permutation symmetry"

More from Research