Liquid AI doubles a model tokenizer from 65K to 128K in place
pmttyji · reddit · 2026-07-22
Liquid AI shared the recipe behind LFM2.5-8B-A1B’s tokenizer upgrade: it expands a pretrained model’s tokenizer in place rather than retraining from scratch.
Key points:
- Vocabulary was doubled from 65K to 128K.
- The goal was to fix languages that the original tokenizer segmented too aggressively.
- The post links to both a blog explanation and an arXiv technical report, making this a concrete method paper rather than just a model release.
The accompanying figure shows the tokenizer comparison on English and Thai examples, illustrating that the expanded tokenizer can preserve sequence length while improving segmentation quality for languages like Thai.
More from Research
- Artificial Analysis breaks down all sub-evaluations in Intelligence Index v4.3.2 — ArtificialAnlys · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27