Why tokenizer optimization is hard: expensive pretraining, slow feedback, and non-differentiable design
paul_cal · x · 2026-07-29
The author explains why text token representation hasn't been hyper-optimized: 1) Pretraining is slow and expensive; OpenAI has done only a handful of pretrains since GPT-3. Deficiencies are only legible at scale, creating a slow feedback cycle. 2) Tokenizer design is not gradient-descent friendly; it's a symbolic program that can't be easily learned end-to-end with the language model. Changing tokenizers is perilous and costly. The transformer needs a stable base, and tokenizers are currently trivial CPU programs with no additional inference cost.
Related event: High Pre-training Costs Hinder Tokenizer Optimization(2 posts)→
More from Research
- Former Meta researcher launches a stealth physical-AI lab focused on interpretability — soniajoseph_ · 2026-07-29
- World Labs argues the key robotics metric is sim-to-real success rate alignment — andrew_n_carr · 2026-07-29
- Interactive rings may need haptic output, not just voice input, for AI use — plopesresearch · 2026-07-29
- A 300-page Princeton thesis maps RL’s shift from games to world models — udmrzn · 2026-07-29
- PostTrainBench v1.1 flags 234 contaminated runs and tightens anti-cheat rules — scaling01 · 2026-07-29
- Nvidia’s LatentMoE is already shaping MoE pretraining after a paper from six months ago — peterjliu · 2026-07-29