Effective learning rate explains peak-then-decay schedules as bias-variance optimum

burny_tech · x · 2026-10-28

A blog post unifying recent findings in pretraining optimization dynamics:

Two empirical phenomena: Hyperball pins the Frobenius norms of weight matrices and their updates to constants, yielding 20–30% token-equivalent speedup over weight decay; and optimal learning-rate curves from multiple independent studies all show a peak-then-decay shape.
Key physical quantity: networks with normalization layers are scale-invariant, so progress is governed by the effective learning rate η = η·‖U‖/‖W‖ (angular update size). Growing weight norms impose an implicit decay on top of any nominal schedule — Hunyuan's ELR experiments show WSD/cosine/linear schedules with nominal peaks differing 2.4× produce nearly identical angular-update curves after 1000 steps.
Shape criterion: peak-then-decay is the optimum of the bias–variance trade-off — large early steps cut bias fast, decay-to-zero late suppresses gradient variance; schedules within the optimal family perform equivalently.

Takeaway: the effective learning rate is the true driver of training, and bias–variance trade-off dictates its geometry.

More from Research