Frozen 16 MB vocabulary slashes small LLM size to 46 MB with no quality loss
pmttyji · reddit · 2026-10-21
An arXiv paper argues the embedding table dominates small LLMs: in Gemma 3 270M it holds 168M of 268M parameters and stays 16-bit after quantization.
Method: replace it with a frozen 16 MB table of 512-bit codes (PCA projection + sign binarization, zero trainable params), then train the same Gemma-shaped body from scratch.
Results:
- The learned table costs 2.6x trainable parameters for just 0.20 nats of validation loss; the frozen table matches or beats it on LAMBADA, HellaSwag, PIQA (only ARC-Easy favors learned)
- With a ternary BitNet b1.58 body the whole model is 46 MB, running at 615-740 tokens/s on a laptop CPU in 62 MB memory — 5-6x faster than the float model's 8-bit GGUF at 1/7 the memory
- Frozen table wins 1.7-2.3 nats on rare words; accepts 10,000 new words post-training at 3% cost vs 43% collapse for learned
- Weights, data and all eval answers are open-sourced
More from Infra
- GitHub hit by delays for Pull Requests and Webhook deliveries — DanielLockyer · 2026-10-21
- NVIDIA expands supply chain: substrate capacity to grow 2.5x, Amkor packaging up 300% — nvidia · 2026-10-21
- Node.js 24.22.1 Docker images finally published after hours-long delay — DanielLockyer · 2026-10-21
- Micron pitches co-design memory roadmap as key to meeting AI demand — BenBajarin · 2026-10-21
- Memphis votes to temporarily ban new data center construction for 135 days — Polymarket · 2026-10-21
- Micron details AI-class NAND: high-IOPS and high-bandwidth lines for KV cache and long context — BenBajarin · 2026-10-21