Caveats: single-seed results, no clear win at 405M-1.4B evals

ziv_ravid · x · 2026-11-19

The author discloses limitations: mostly single-seed runs; no help for AdamW with ReLU² at 405M; downstream evals at 405M and 1.4B show no clear win over constant decay; and the reason this step-size trajectory works remains unexplained.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →