Tokenizer Myths and Where To Find Them: Stanford paper debunks three BPB myths
rohanpaul_ai · x · 2026-11-06
Stanford's Rylan Schaeffer and Sanmi Koyejo release a 35-page arXiv paper challenging three myths in tokenization research: (1) BPB is not tokenizer- or vocabulary-size-agnostic—the two factors of tokens-per-byte × bits-per-token don't cancel, and untrained models still score differently by tokenizer; (2) lower BPB doesn't mean a better model—BPB weakly correlates with downstream performance and prefers certain vocabulary sizes even at chance accuracy; (3) 'tokenizer-agnostic' metrics disagree—the unigram-normalized loss's 0.989 correlation with BPB falls to 0.057 after fixing a confounder, and the two metrics select different vocabularies. They formalize a tokenizer trilemma: no metric can score only the true distribution best, give identical predictions identical scores, and tie all untrained models—choosing a metric is choosing a research question.
Related event: Stanford paper: bits-per-byte is not tokenizer-agnostic(3 posts)→
More from Research
- Meta's Axia Learns Scaling Laws for Ranking Models from a Few Trial Runs — _reachsumit · 2026-11-06
- Meta's CAP-Mix Replaces Hard Semantic IDs with Soft Prototype Mixtures for Generative Retrieval — _reachsumit · 2026-11-06
- Stanford team proposes Joules-to-GDP: measuring AI's economic value per joule of energy — niloofar_mire · 2026-11-06
- Weaver: Incremental Reasoning Over Conversation Memory for Personalized RAG — _reachsumit · 2026-11-06
- METR report on AI accelerating AI R&D flips a skeptic: >50% on superintelligence in 18 months — GaryMarcus · 2026-11-06
- Optimal Re-Ranking Depth Varies by Query: LLM-Judged Predictor Cuts Depth 3x — _reachsumit · 2026-11-06