Liquid AI doubles a model tokenizer from 65K to 128K in place
pmttyji · reddit · 2026-07-22
Liquid AI shared the recipe behind LFM2.5-8B-A1B’s tokenizer upgrade: it expands a pretrained model’s tokenizer in place rather than retraining from scratch.
Key points:
- Vocabulary was doubled from 65K to 128K.
- The goal was to fix languages that the original tokenizer segmented too aggressively.
- The post links to both a blog explanation and an arXiv technical report, making this a concrete method paper rather than just a model release.
The accompanying figure shows the tokenizer comparison on English and Thai examples, illustrating that the expanded tokenizer can preserve sequence length while improving segmentation quality for languages like Thai.
More from Research
- 3D ResNet Paper Crosses 3,000 Citations Eight Years After CVPR 2018 — HirokatuKataoka · 2026-09-11
- Sample selection and ordering matter a lot in LLM training: DataFlex makes data scheduling dynamic — Puzzleheaded_Box2842 · 2026-09-11
- Jeff Heaton's Intro to the Math of Neural Networks eBook Is Free to Download — blaizedsouza · 2026-09-11
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11