Why tokenizers still resist end-to-end optimization despite years of pretraining
seanmcdonaldxyz · x · 2026-07-29
Paul Cal argues that text token representation has not been heavily optimized because the feedback loop is too slow and expensive: frontier pretraining is rare, and bad tokenization choices may only show up at scale.
He also notes that tokenizers are fundamentally awkward to optimize end-to-end:
- they are symbolic programs, not just learned embeddings;
- they define the output objective and shape what representations are even possible;
- changing them later is hard to backport and may not improve results;
- making them more complex risks turning them into another expensive model layer.
His conclusion: there is still no established, frontier-viable end-to-end tokenizer optimizer that can simultaneously minimize tokens and preserve performance across tasks.
Related event: High Pre-training Costs Hinder Tokenizer Optimization(2 posts)→
More from Research
- Former Meta researcher launches a stealth physical-AI lab focused on interpretability — soniajoseph_ · 2026-07-29
- World Labs argues the key robotics metric is sim-to-real success rate alignment — andrew_n_carr · 2026-07-29
- Interactive rings may need haptic output, not just voice input, for AI use — plopesresearch · 2026-07-29
- A 300-page Princeton thesis maps RL’s shift from games to world models — udmrzn · 2026-07-29
- PostTrainBench v1.1 flags 234 contaminated runs and tightens anti-cheat rules — scaling01 · 2026-07-29
- Nvidia’s LatentMoE is already shaping MoE pretraining after a paper from six months ago — peterjliu · 2026-07-29