UTF-8 implicitly favors Latin scripts: Hindi words cost 4x the bytes of romanization
rdesh26 · x · 2026-11-08
While rereading the H-Net paper by @albertgu et al., the author observed that byte-level models implicitly favor Latin-script languages due to UTF-8's encoding order.
- Latin "A" (U+0041) is 1 byte, while Devanagari "क" (U+0915) takes 3 bytes.
- The morphological hit is subtler: Hindi consonants carry an implicit vowel (क = k + a) that drops in conjuncts, but UTF-8 encodes "क + ्" as 6 bytes; the word "क्या" ("what") costs 12 bytes vs 3 for its transliteration "kya".
The author argues the root problem lies in the digital encoding of these languages, implying byte-level models face systematically harder learning for non-Latin scripts.
More from Research
- 159-page open course notes on e3nn equivariant GNNs released with 60 AI-tutored exercises — burkov · 2026-11-08
- Researchers reverse-engineer iPhone keyboard's hidden blocklist: ~100K phrases across 28 languages — sethlazar · 2026-11-08
- Student-built ApexW optimizer speeds up SD 1.5 LoRA training 6.3% on a free T4 — Ready_Parking_5435 · 2026-11-08
- Reinforcement Learning: Foundations textbook by TAU/Technion researchers now in print, draft free online — AvivTamar1 · 2026-11-08
- Pairing two papers: subliminal distillation baggage and RL's pass@k ceiling — andrew_n_carr · 2026-11-08
- A few labeled examples can steer generative models to cover regions of interest and skip the rest — caglar_ee · 2026-11-08