Small-scale proxies reproduce large-scale Transformer training instabilities
stochasticchasm · x · 2026-08-28
The thread discusses the arXiv paper "Small-scale proxies for large-scale Transformer training instabilities" (Wortsman et al.). Key finding: training instabilities seen in large models (attention logit growth, output-logit/log-probability divergence) are expensive to reproduce at scale, but the authors show they also appear in small models trained at high learning rates, with large-scale mitigations equally effective there. The paper systematically studies how warm-up, weight decay, and other interventions affect sensitivity of final loss to learning rate. The discussion adds that batch size warmup didn't seem to help, and results may not transfer across architectures.
More from Research
- NVIDIA Details QAD Pipeline for Optimizing Nemotron Model — PyTorch · 2026-08-28
- Study: AI Coding Agents Are Reshaping PR Vocabulary — ycombinator · 2026-08-28
- Netflix paper: production LLM judges need a lifecycle, not one-time validation — rohanpaul_ai · 2026-08-28
- AI Book Club to host live chat with author of 'Build a Reasoning Model' — sophiamyang · 2026-08-28
- Tech Comparison: Sparse Attention Implementation in Qwen vs. Minimax — stochasticchasm · 2026-08-28
- Analysis compares Qwen and MiniMax sparse attention implementations — stochasticchasm · 2026-08-28