Paper reveals LM Head as gradient bottleneck amid Anthropic vocab size rumors
JFPuget · x · 2026-08-15
Addressing rumors about Anthropic's model vocabulary size, this post cites the paper "Lost in Backpropagation: The LM Head is a Gradient Bottleneck". The paper argues that projecting output features to the vocabulary dimension creates not just an expressivity bottleneck, but an optimization bottleneck. Backpropagating V-dimensional gradients through a rank-D linear layer causes unavoidable compression, altering training feedback for most parameters. Empirical measurements show 95-99% of gradient norm is suppressed by the output layer, leading to suboptimal update directions.
Related event: Paper: LM Head Bottleneck Suppresses 95-99% of Gradients(3 posts)→
More from Models
- Google's Gemini 4 Argon performs noticeably worse in AI Mode than in AI Studio — gaganghotra_ · 2026-10-16
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- ChatGPT UI screenshot reveals Wallet coming soon plus Email, Calendar, Health and Finances integrations — jxnlco · 2026-10-16
- Why the next OpenAI and Anthropic models may still disappoint: no native audio or video input — imjustnewatai · 2026-10-16
- Thousands Hit by ChatGPT Standard Voice Mode Regression: Wrong Voice, Interruptions — shaunralston · 2026-10-16
- Gemini Argon 4 Quota Exhausted, Reset Countdown Shows Nearly 7 Days — ssh4net · 2026-10-16