Continued BPE training improves tokenizer extension and pruning across model families
burkov · x · 2026-07-25
Tokenizer adaptation matters when moving pre-trained language models to new domains or languages. The paper focuses on two complementary operations: vocabulary extension and pruning.
- Instead of training a brand-new tokenizer and appending only non-overlapping tokens, the authors propose continued BPE training: keep learning BPE merges on new data and extend the existing tokenizer.
- Across multiple languages and model families, they report better tokenization efficiency and improved utilization of the added vocabulary.
- They also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality.
- The authors release an open-source toolkit for controlled vocabulary modification.
More from Research
- Chinese Room argument resurfaces in the modern LLM understanding debate — Roger_M_Taylor · 2026-07-25
- Jetson-PI: Boosting Onboard VLA Control Frequency to 6Hz, 8.6x Faster — 机器之心 · 2026-07-25
- Nature paper turns pathologists’ slide-viewing behavior into training data for AI agents — yuyinzhou_cs · 2026-07-25
- Surya Ganguli says physics may be a better route to understanding AI than theorem-proof theory — SuryaGanguli · 2026-07-25
- ARC-AGI-4 should stay private, after Opus 5 scored 3× the next-best model on ARC-AGI-3 — burny_tech · 2026-07-25
- Compiled by Claude: A Comprehensive List of NeurIPS Workshops — DanielKhashabi · 2026-07-25