Vocabulary Tax paper trims 89.7% of unused LLM tokens for on-device use
kalyan_kpl · x · 2026-11-19
The paper "The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use" cuts LLM memory footprint by removing unused vocabulary entries while keeping the tokenizer correct.
- Problem: Multilingual LLMs carry vocabularies up to 262K tokens; embedding tables, output heads and per-layer embeddings consume memory even for tokens a deployment never uses.
- Two trimming strategies:
- Script-based: keeps tokens of the target languages' writing systems, preserving tokenization exactly.
- Corpus-based: keeps only tokens observed in deployment-specific text — larger memory savings, but tokenization may change.
- Results: Across six models (270M–3B params) and 11 languages, a median of 89.7% of the vocabulary is removed.
More from Infra
- KAIST Proposes PROLA: Low-Rank Adaptation for Spatiotemporal Weather Downscaling — MLVLab · 2026-11-19
- Bypassing PyTorch with HIP kernels makes MiniMax H3 up to 3x faster on AMD R9700 — Apprehensive_Sky892 · 2026-11-19
- Ex-DeepSeek core member open-sources Linger, a local AI agent claiming 5x faster inference — Aiden_Tech_Ai · 2026-11-19
- China opens first overseas-targeted AI cluster on Huawei Ascend chips — pstAsiatech · 2026-11-19
- Tencent says US chip curbs 'unfortunate', may slow AI training short-term — pstAsiatech · 2026-11-19
- Samsung dismisses AI bubble fears: neoclouds are just 8% of the market, chip shortage to persist — BenBajarin · 2026-11-19