A frozen 16MB vocabulary replaces learned embeddings in small and ternary LMs
Sai Kiran Bathula · hf · 2026-10-19
An independent research paper challenges a small-LM design assumption: the embedding table is overpriced. In Gemma 3 270M, the embedding holds 168M of 268M parameters and stays 16-bit under most quantization, dominating file size once the rest is compressed.
Method and results:
- Derive a frozen table of 512-bit codes (16MB, zero trainable params) from a pretrained embedding via PCA projection and sign binarisation, then train the same Gemma-shaped body from scratch on 2.05B tokens with and without it
- The learned table buys only 0.20 nats of validation loss for 2.6x the trainable parameters
- The frozen-table model is better or tied on LAMBADA, HellaSwag, and PIQA; only ARC-Easy favors the learned version
- With a ternary BitNet b1.58 body, the whole model is a 46MB file, still tying the float model on LAMBADA and PIQA, with faster CPU inference
Takeaway: embedding layers in small models are a major compression target, with clear implications for on-device deployment.
More from Infra
- Alibaba Cloud shows Omarchy running on Wuying AI cloud PCs for enterprise agents — AIFlow_ML · 2026-10-19
- NVIDIA paper shows cross-model KV cache transfer skipping prefill, 2.7–25x faster than recomputing — solyarisoftware · 2026-10-19
- pi-durable v1.3.0 adds SQLite table prefixes, fixes 140 MB memory spike on DO — solyarisoftware · 2026-10-19
- Open-source SwiftUI dashboard brings a Docker Desktop-like UI to Apple's native container runtime — AIFlow_ML · 2026-10-19
- Strata runs on Windows with dual AMD GPUs despite no official support: ~1350pp on a 9070XT — Ok-Importance-3529 · 2026-10-19
- Thinking Machines explains LLM nondeterminism and ships batch-invariant kernels for reproducibility — AccBalanced · 2026-10-19