Factorised output heads let tiny LMs use a 131k vocab at equal training FLOPs
BLCNYY · x · 2026-11-16
An independent study (Jakub Michałek, code and logs open-sourced) compares three output-layer designs for small LMs (6 blocks, d=384, 62–108M params, trained on FineWeb-Edu) at equal training FLOPs.
Key findings
- With a 131k vocabulary at d=384, a flat softmax head consumes 79.5% of total training FLOPs.
- Both factorised heads beat the flat head: a two-level head (8,192 frequent tokens with own logits + 4,096 k-means clusters) and adaptive softmax reach -0.0095 / -0.0092 BPB for 131k vs the best 50k model, while the flat head degrades by +0.0495.
- The counterintuitive twist: adaptive softmax's usual narrow tail hurts. Scoring the tail at full width improves BPB by -0.0107 / -0.0093 and ties the two-level head (-0.0010 / +0.0004 under a pre-registered ±0.003 tie rule).
- The two-level head trains at 1.98× (50k vocab) to 4.29× (131k vocab) the tokens per second of the flat head on the author's GPU.
Caveats: only 1–2 seeds per setting; the 131k BPE was trained on the corpus while 50k is GPT-2's tokenizer, so vocab size and domain change together. Treat conclusions as preliminary.
More from Research
- Chollet: reasoning chain generation is search, not interpolation — teortaxesTex · 2026-11-16
- Terence Tao: math has moved from proof scarcity to proof abundance — haider1 · 2026-11-16
- Wrong AI advice hurts doctors' judgment more than right advice helps: study of 16,641 decisions — ValerioCapraro · 2026-11-16
- Free online book 'The Science of Computing' links PDEs, numerics, and cache performance — burkov · 2026-11-16
- Controlled speech experiments: joint training from scratch always beats separate pretraining — giffmana · 2026-11-16
- Free Textbook: Algebraic Topology for Data Scientists and TDA — burkov · 2026-11-16