Liquid AI expands a pretrained tokenizer from 65K to 128K without retraining from scratch
JosephJacks_ · x · 2026-07-21
Liquid AI shares a technical report on in-place tokenizer expansion for pretrained LLMs, showing how to upgrade a tokenizer without starting pretraining from scratch.
What they changed
- Expanded the tokenizer from 65K to 128K vocabulary.
- Reused the original BPE merges, then continued BPE training on a multilingual corpus.
- Initialized new embeddings from the model’s existing knowledge and adapted in two stages: embedding-only training followed by full continued retraining.
Why it matters
- On LFM2.5-8B, the new tokenizer cuts token counts for underrepresented languages, including Thai (3.7× faster decode), Bengali (3.1×), and Vietnamese (2.4×).
- Their report frames this as a way to improve on-device efficiency without discarding prior pretraining compute.
More from Infra
- PyTorch details 2.12x faster on-device streaming ASR on Arm SME2 via ExecuTorch — PyTorch · 2026-10-27
- Revenue per GPU can rise as token prices fall, argues Gavin Baker — GavinSBaker · 2026-10-27
- Anthropic S-1: only $19B total compute spend since inception, overbuild warning — bookwormengr · 2026-10-27
- CoreWeave, You.com and NVIDIA lift BrowseComp 22.9% in 3-week RL post-training sprint — RichardSocher · 2026-10-27
- Grid queues make power architecture an AI-factory decision; Exowatt and Redwood offer fixes — ingliguori · 2026-10-27
- Two CVSS 9.9 Wasmtime flaws allow sandbox escape, one ARM64-only since 2020 — basedjensen · 2026-10-27