Paper: LM Head is a Gradient Bottleneck, losing 95-99% of gradient norms
tokenbender · x · 2026-08-16
A paper submitted to COLM'26 reveals that the final layer of Language Models (LMs) is not just an expressivity bottleneck but also an optimization bottleneck. Projecting D-dimensional outputs to V-dimensional logits (where D << V) causes unavoidable gradient compression during backpropagation.
Key Findings:
- Severe Gradient Loss: Empirical measurements show that 95-99% of gradient norms are suppressed by the output layer, leading to suboptimal update directions.
- Training Inefficiency: This flaw makes trivial patterns unlearnable and drastically affects LLM training dynamics.
- Architecture Agnostic: The inherent design flaw causes training inefficiencies at scale regardless of the model architecture.
Related event: Paper: LM Head Bottleneck Suppresses 95-99% of Gradients(3 posts)→
More from Research
- Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change? — Sauers_ · 2026-10-16
- Researcher credits GPT for multiple accepted papers, incl. disproving Wegner's 1965 conjecture — ctjlewis · 2026-10-16
- Open-source shopper attention heatmap: YOLO11-pose + Gaze-LLE + Depth Anything V2 — claudioe · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks — maier_ak · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16