LLM tokenizers repeat the 1870s Chinese telegraph problem: up to 17x cost gap across languages
demian_ai · x · 2026-11-10
- History: Morse code was built around English—"E" is a single dot. The 1871 Chinese telegraph code assigned each character a four-digit number, each digit costing five dots-and-dashes, so one character took 20 signals; a 25-word message took 30 minutes to encode vs 2 minutes in English.
- Today: LLMs bill by tokens, and tokenizers learned their chunks mostly from English/Latin web text, so other languages get chopped finer. Among the top 20 online languages, the priciest needs 1.5–5x the tokens of the cheapest; across 200 languages one model's gap reached 17x.
- Knock-on effects: More tokens mean less fits in context and slower responses. Script hurts more than language rarity—Burmese, Tibetan, Khmer and Lao fare worst.
Takeaway: every network is priced in the units of whoever built it, and speakers of other units pay the difference until someone redesigns the code.
More from AGI Musings
- Report: AI agent incident logs sit with the culprits — Walrus proposes tamper-evident onchain records — 0xSammy · 2026-11-11
- Allie Miller: AI already shows judgment — you trust it every day without noticing — alliekmiller · 2026-11-11
- Study: 9 of 10 AI Agents Overrode a 1-Cent Daily Spending Cap on Their Own — JakRenden2 · 2026-11-11
- Nat Lambert: human expertise to diffuse AI's benefits is vastly underestimated — natolambert · 2026-11-11
- Mat Dryhurst: screen-free digital connection is uniquely enabled by AI — matdryhurst · 2026-11-11
- LinkedIn: AI created 2M jobs; Andrew Ng pitches embedding AI engineers in business teams — DeepLearningAI · 2026-11-11