A new tokenizer-adaptation method cuts Thai tokens 4× on pretrained models
kastnerkyle · x · 2026-07-23
Researchers describe a practical way to adapt the tokenizer of a pretrained model without retraining from scratch. Instead of adding unused tokens through the usual extension route, they continue BPE learning on new data and pair it with leaf-based vocabulary pruning.
Reported gains include:
- Thai: 4.0× fewer tokens
- Vietnamese: 2.6× fewer tokens
- Hindi: 2.4× fewer tokens
- Estimated 2.2× to 3.7× faster per-character decoding on-device for these languages
The paper argues that tokenizers are under-studied despite strongly affecting multilingual model usage, and the authors release the method as an open-source toolkit.
More from Models
- OpenAI ships GPT-6.1 Terra, but Claude Sonnet 5.5 beats it on Terminal-Bench 4.0 — imjustnewatai · 2026-10-27
- We Built a 506-Race Election Forecast, Then Had Claude, ChatGPT and Grok Call Every Close Race — justin_hart · 2026-10-27
- Chart: GPT-6.1 Terra beats Sol max on cost, and max variant outperforms Astra max — GabGarrett · 2026-10-27
- KOL defends B-tier rating for GPT-6.1 Sol, blames OpenAI's botched release naming — Angaisb_ · 2026-10-27
- php2shell exploit benchmark: only 2 of 5 frontier models can port the bug — xeophon · 2026-10-27
- Qwen Code v0.26.2-preview.0 adds session permission rules and experimental shared agent loop — qwen-code-review-bot · 2026-10-27