Liquid AI Expands LFM2 Tokenizer to 128K In-Place, Boosting Multilingual Decoding

Liquid AI released a technical report and blog post detailing an in-place tokenizer expansion for their pretrained model LFM2.5-8B-A1B. This method successfully scales the vocabulary from 65K to 128K without retraining the model from scratch, aiming to fix the previous tokenizer's issue of over-segmenting certain languages and thereby improving multilingual processing efficiency.

Confirmed

According to summaries by @pmttyji and @kastnerkyle, the research team did not simply hard-add new words to the vocabulary. Instead, they continued Byte-Pair Encoding (BPE) learning and applied leaf-based vocabulary pruning to remove redundant tokens. This in-place expansion method effectively balances the increased vocabulary size with the retention of the original model's knowledge.

Why it matters

The expanded tokenizer has delivered immediate results in multilingual support. @maximelabonne noted that for Thai, the optimized tokenizer reduces the token count by approximately 4 times. Furthermore, doubling the vocabulary did not slow down performance; instead, it achieved up to a 3.7x increase in decoding speed on edge devices, demonstrating high practical engineering value.

2026-07-21 ~ 2026-07-23 · 5 related posts

• [source] LFM2 tokenizer expansion cuts Thai tokens 4× and speeds on-device decoding up to 3.7× — maximelabonne · 2026-07-21
• Liquid AI expands a pretrained tokenizer from 65K to 128K without retraining from scratch — JosephJacks_ · 2026-07-21
• LFM2.5-8B-A1B doubles its tokenizer vocab and cuts on-device decoding time up to 3.7x — maximelabonne · 2026-07-22
• [source] Liquid AI doubles a model tokenizer from 65K to 128K in place — pmttyji · 2026-07-22
• A new tokenizer-adaptation method cuts Thai tokens 4× on pretrained models — kastnerkyle · 2026-07-23