Hugging Face argues there is no truly tokenization-free LLM
cjmaddison · x · 2026-07-29
Tokenization is still the real issue in LLM design
A Hugging Face post argues that calls to “get rid of tokenizers” miss the point: even UTF-8 byte approaches still make design choices, so there is no truly choice-free tokenization scheme.
The image also pushes back on common complaints that are often blamed solely on tokenizers — spelling, reversal tasks, non-English performance, arithmetic, Python code, early stops at </endoftext>, and whitespace handling — and says tokenizers are often part of the solution rather than only the problem. It points to work on identifying and removing problematic “glitch tokens” before training, instead of trying to eliminate tokenization entirely.
More from Research
- Bagel Labs releases WorldDiT, a sub-billion-parameter robot control model — EXM7777 · 2026-07-29
- Why LLMs still fail at chess may be a key test of how capabilities emerge — alex_peys · 2026-07-29
- Anthropic's Claude Halves Post-Quantum Security, Accelerates AES Attacks by 800x — petrusenko_max · 2026-07-29
- Tech Report Insight: External Model Init for Vision Encoders Falls Short — SeunghyunSEO7 · 2026-07-29
- AI drug discovery pipeline now spans small molecules and nanobodies in lab tests — const_reborn · 2026-07-29
- Jon Durbin says he pre-trained a 20B MoE for under $10 an hour — const_reborn · 2026-07-29