LFM2 tokenizer expansion cuts Thai tokens 4× and speeds on-device decoding up to 3.7×
maximelabonne · x · 2026-07-21
A tokenizer expansion recipe aims to add languages to LFM2
A new blog post explains how the team expanded LFM2's tokenizer to support additional languages more efficiently.
The quoted example says LFM2.5-8B-A1B doubled its tokenizer size from 65K to 128K to stop some languages from being split too finely. The reported gains were substantial:
- Thai: 4.0× fewer tokens
- Vietnamese: 2.6× fewer tokens
- Hindi: 2.4× fewer tokens
- estimated 2.2× to 3.7× faster per-character decoding on-device for those languages
The post focuses on the recipe for upgrading a pretrained model's tokenizer in place.
More from Research
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27
- Community-trained MOSS Sharpness Slider LoRA: 72 photos, 500 steps, +/-1 focus control — aoleg77 · 2026-10-27
- CoreWeave, You.com and NVIDIA lift BrowseComp 22.9% in 3-week RL post-training sprint — RichardSocher · 2026-10-27