"Tokenizer Myths and Where to Find Them": talk slides debunk tokenizer folklore
RylanSchaeffer · x · 2026-10-03
Researcher RylanSchaeffer released the slides for his talk "Tokenizer Myths and Where to Find Them," systematically debunking common misconceptions about tokenizers. A preprint is forthcoming and feedback is welcome.
A later part of the thread also introduces a theoretical result — the tokenizer trilemma: any evaluation metric can satisfy at most two of three reasonable desiderata, so choosing a metric implicitly constrains what research is possible.
Related event: Researcher Debunks Tokenizer Myths, Proves Tokenizer Trilemma(3 posts)→
More from Research
- MIT's Cathy Wu boosts RL training efficiency 30x to design smarter transportation systems — MIT News AI · 2026-10-03
- Economist calls for mirror life regulations; researchers say the threat is wildly premature — anshulkundaje · 2026-10-03
- Benchmarking long-range ML interatomic potentials on SrTiO3 defect energetics — rbhar90 · 2026-10-03
- Meta AI's RL-XAR fixes AI slop via expert-aligned rubrics, plus 4 more papers explained — idanbeck · 2026-10-03
- Neuralink pretrained AI models on 50,000 hours of unlabeled brain data, betting on the scaling playbook — Dr_Singularity · 2026-10-03
- Minerva language model deployed on Proto MCP to revisit RNA annotations with ChatGPT — BrianHie · 2026-10-03