Researcher Debunks Tokenizer Myths, Proves Tokenizer Trilemma

AI researcher Rylan Schaeffer published slides from his talk debunking common tokenizer myths, proving a 'tokenizer trilemma' where only two of three reasonable metrics can be satisfied. He also argued that Bits-per-Byte (BPB) only measures encoding length and has flaws as an evaluation metric.

2026-10-03 ~ 2026-10-03 · 3 related posts

• "Tokenizer Myths and Where to Find Them": talk slides debunk tokenizer folklore — RylanSchaeffer · 2026-10-03
• Tokenizer trilemma: BPB metric measures coding length and nothing else, thread argues — RylanSchaeffer · 2026-10-03
• Researchers prove a "tokenizer trilemma": metrics can satisfy only two of three desiderata — RylanSchaeffer · 2026-10-03