95-99% of Gradient Norm Lost at LM Head; Raising Head Rank 32→4096 Speeds Convergence 16x
yoavartzi · x · 2026-11-13
The cited paper shows that 95–99% of the gradient norm is lost at the LM head during backpropagation. The authors demonstrate the softmax bottleneck (hidden dim D much smaller than vocab V) hurts optimization, not just expressivity: raising head rank from 32 to 4096 made a 2B model's pretraining converge 16x faster. The commenter adds CPLM can mitigate the issue to some degree.
More from Research
- Qwen3.5-4B fine-tuned in 27.5 min on one GPU lifts MedQA accuracy to 74.5% — MaziyarPanahi · 2026-11-13
- AromaGen transmits smells in real time with AI, hitting 71% in blind tests — pliang279 · 2026-11-13
- Muon shown to be a special case of Shampoo optimizer — _arohan_ · 2026-11-13
- BioSecBench-Structure: a benchmark for agentic molecular structure reasoning — kenbwork · 2026-11-13
- One 8×B200 node serves 3,000 DeepSeek agents, 19x official capacity — james_y_zou · 2026-11-13
- DocFS turns documents into file systems so VLM agents can reason over them — _reachsumit · 2026-11-13