Survey by 30+ authors maps modern NLP tokenization, from subword wins to its flaws
rdesh26 · x · 2026-11-09
A comprehensive survey, Tokenization: A Survey for Modern NLP (Marco Cognetta et al., #6 trending on alphaXiv this September), systematizes the first hidden step of every LLM pipeline.
- Main approach: subword tokenization dominates — larger granularity than chars/bytes, smaller than words. Versus word-level models it solves the OOV problem while capping vocabulary size, cutting embedding parameters and logits; versus character-level models it packs more information per token and yields much shorter sequences.
- Known weaknesses: rigid segmentation schemes that don't generalize across languages; segmentation ambiguities enabling adversarial attacks; explicit balancing required for token allocation between high- and low-resource languages in multilingual settings.
- The survey also covers recent script-specific encoding ideas, making it a solid reference for anyone working on tokenization.
More from Models
- Toby Ord pushes back: reasoning and coding models aren't separate eras — tobyordoxford · 2026-11-09
- 'You'd be surprised how many are my coworkers': the odd anti-local-LLM crowd sparks debate — TheZachMueller · 2026-11-09
- System One models are back: decision/extraction APIs cut insurance claim flow from 61.7s to 19.3s — max_gladysh · 2026-11-09
- Community brands Opus 5 as a rare LLM release worse than its predecessor — adonis_singh · 2026-11-09
- Mistral launches first in 100% of 59 nuclear dilemma games while GPT, Claude, Grok, Kimi all cooperate — eliebakouch · 2026-11-09
- Dev says Opus 5.5 was the best model release of the year — adonis_singh · 2026-11-09