Claude's tokenizer has only 16K vocabulary, sparking speculation about softmax bottleneck
rickasaurus · x · 2026-08-14
Developers have discovered that the latest Claude models have a tokenizer vocabulary of only about 16K, far lower than Qwen3.5's 250K. This anomaly has sparked discussion, with theories suggesting Anthropic may be trying to remove the softmax bottleneck by reducing vocabulary size. A reproduction project confirmed that Claude 4.7+ has an even lower estimate of just 15K entries.
More from Models
- Google drops Gemini 3.7 Flash, developers build 10 amazing apps within 24 hours — CodeByPoonam · 2026-08-14
- Gemini 3.7 Flash Tested: Builds Lunar Lander Game in 54 Seconds — TawohAwa · 2026-08-14
- Gemini 3.7 Flash vs Grok 4.6: 3D Wright Flyer Showdown — TawohAwa · 2026-08-14
- Gemini 3.7 Flash vs Opus 5: Cost and Speed Compared — TawohAwa · 2026-08-14
- Gemini Flash 3.7 UI Design Impresses: $3.4 Builds 3D Skate Game — TawohAwa · 2026-08-14
- Gemini 3.7 Flash Much Faster Than 3.6 Flash in Test — TawohAwa · 2026-08-14