Paper: LM Head is a Gradient Bottleneck, losing 95-99% of gradient norms
tokenbender · x · 2026-08-16
A paper submitted to COLM'26 reveals that the final layer of Language Models (LMs) is not just an expressivity bottleneck but also an optimization bottleneck. Projecting D-dimensional outputs to V-dimensional logits (where D << V) causes unavoidable gradient compression during backpropagation.
Key Findings:
- Severe Gradient Loss: Empirical measurements show that 95-99% of gradient norms are suppressed by the output layer, leading to suboptimal update directions.
- Training Inefficiency: This flaw makes trivial patterns unlearnable and drastically affects LLM training dynamics.
- Architecture Agnostic: The inherent design flaw causes training inefficiencies at scale regardless of the model architecture.
Related event: Paper: LM Head Bottleneck Suppresses 95-99% of Gradients(3 posts)→
More from Research
- IBM Study: Benchmark High Scores Depend on Wording, Strong Models More Fragile — omarsar0 · 2026-08-16
- HUSKY Enables Humanoid Robots to Skateboard via Physics-Aware Control — tom_doerr · 2026-08-16
- ArchAgent v2 achieves multi-level data prefetching via evolutionary search — dair_ai · 2026-08-16
- LLMRouter open-sources 16+ implementations with xRouteBench for LLM routing — Justgototheeffinmoon · 2026-08-16
- New method: replay partial reasoning traces to train student models without live environment — YouJiacheng · 2026-08-16
- LLMs Struggle with Complex Structured Outputs; Developer Shares 'Researcher-Formatter' Split Architecture — DiligentBend9150 · 2026-08-16