LLM Architecture: How Large Vocabularies Mitigate Softmax Rank Bottlenecks
kalomaze · x · 2026-08-14
Researcher @kalomaze discussed the Softmax rank bottleneck issue in large language model architectures like DeepSeek V3.
- Context: In a reply, it was noted that models seem heavily designed to avoid the softmax rank bottleneck, otherwise some nats can objectively never be modeled correctly.
- Analysis: He pointed out that DSV3-like backbones have hidden dimensions of around 7168. With a 15k vocabulary size, the softmax is 10x less rank-bottlenecked in terms of the finegrainedness of the learned probability distribution compared to narrower architectures.
More from Research
- Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change? — Sauers_ · 2026-10-16
- Researcher credits GPT for multiple accepted papers, incl. disproving Wegner's 1965 conjecture — ctjlewis · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks — maier_ak · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16
- Score drift in RL fine-tuning is distillation toward a moving teacher, explains researcher — nrehiew_ · 2026-10-16