DeepMind paper: 5-second-per-layer quadratic simulations predict LLM training spikes
burny_tech · x · 2026-10-25
A DeepMind paper by Ran Tian and Courtney Paquette shows that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate the mysterious loss spikes seen during LLM pretraining.
- Introduces a layer-wise simulation framework that synthesizes stochastic gradients from a checkpoint, avoiding repeated full-network training runs.
- Case study on a 1.3B Transformer: the simulation correctly predicts spiky behavior for an Adam configuration with β2 near 1, at 5 seconds per layer on CPU.
- They also derive a β2 schedule from observed metrics that empirically improves performance.
- The paper notes spikes are sensitive to initialization seed and hard to reproduce — removing outlier data only partially helps, suggesting single data points usually aren't the cause.
More from Infra
- Survival LLM: Free Offline AI Assistant Bundling 1.5M Emergency Articles, No Internet Needed — burny_tech · 2026-10-25
- SGLang vs vLLM: the inference framework race has no weekends — AccBalanced · 2026-10-25
- CASPA summit: chip-to-chip interconnect eats 60-70% of package energy, AI chip design targets 9-month cycles — ai · 2026-10-25
- 8 mixed GPUs plus 256GB DDR4 vs DGX Spark: is this MoE inference rig viable? — Ambitious_Fold_2874 · 2026-10-25
- Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale — blaizedsouza · 2026-10-25
- Memory is 70-80% of AI rack silicon, and Rasgon asks what if prices never collapse — firstadopter · 2026-10-25