Liquid AI expands a pretrained tokenizer from 65K to 128K without retraining from scratch
JosephJacks_ · x · 2026-07-21
Liquid AI shares a technical report on in-place tokenizer expansion for pretrained LLMs, showing how to upgrade a tokenizer without starting pretraining from scratch.
What they changed
- Expanded the tokenizer from 65K to 128K vocabulary.
- Reused the original BPE merges, then continued BPE training on a multilingual corpus.
- Initialized new embeddings from the model’s existing knowledge and adapted in two stages: embedding-only training followed by full continued retraining.
Why it matters
- On LFM2.5-8B, the new tokenizer cuts token counts for underrepresented languages, including Thai (3.7× faster decode), Bengali (3.1×), and Vietnamese (2.4×).
- Their report frames this as a way to improve on-device efficiency without discarding prior pretraining compute.
More from Infra
- Qwen3-8B gets a KV-approximation add-on that halves prefill time without touching the model — teortaxesTex · 2026-09-11
- Burning through two ChatGPT resets a day, user coins the "Huang-Altman Law" — yihui_indie · 2026-09-11
- OpenRouter agents now out-consume humans as AI usage arrives in three waves — AccBalanced · 2026-09-11
- Nvidia Is Now Core to Every Major Robotaxi Stack at Commercial Scale — pdamodaran · 2026-09-11
- 12 KV Cache Reduction Techniques Every AI Engineer Should Understand, Explained — blaizedsouza · 2026-09-11
- The shadow GPU capacity market is formalizing, with Meta selling excess compute to outside buyers — DavidLinthicum · 2026-09-11